Modeling and preoperative planning for kidney surgery

Transcription

1 Modeling and preoperative planning for kidney surgery A thesis submitted in fulfillment of the requirements for the degree of Master of Science By Refael Vivanti Supervised by Prof. Leo Joskowicz The Selim and Rachel Benin School of Engineering and Computer Science The Hebrew University of Jerusalem Jerusalem, Israel June 29,

2 Acknowledgments I would like to thank my supervisor, Prof. Leo Joskowicz, who supported and guided me through the stages of research and the writing of this thesis. I also thank my colleague Achia Kronman, at the Computer-Aided Surgery and Medical Image Processing Laboratory, and all the other members of the laboratory, who were always helpful and informative. I thank Dr. Yoav Mintz from the Hadassah Hebrew University Medical Center, Ein Karem, Jerusalem, for providing the CT data sets and for his advise. I would like to thank also my wife, Maia, for her invaluable support. 2

3 Abstract Kidney surgery is a common and complex procedure. The procedure is required to remove kidney tumors and to treat a variety of renal malfunctions. Nowadays, a minimally invasive approach is the method of choice for partial nephrectomy operations. This procedure presents many challenges to the surgeon stemming from the kidney anatomy and the nature of the minimal invasive technique. Preoperative planning based on Computed Tomography (CT) scans is a crucial step, since it can reduce the risk of the procedures and decrease their duration. Many of the surgical decisions can be pre-determined in the preoperative planning stage. Today, surgeons browse through CT slices, construct a mental representation of the kidney, its internal components and their spatial relations, and plan the surgery accordingly. This approach is often time-consuming, incomplete, and inaccurate. Recently, preoperative planning systems for a variety of procedures have been developed. The systems rely on the automatic generation of patient-specific 3D anatomical models from CT scans. Segmentation methods have been developed for a variety of organs, such as the heart and the liver, and for blood vessels. However, relatively few works address kidney and kidney components segmentation and preoperative planning for partial nephrectomy. This thesis describes semi-automatic methods for the creation of patient-specific kidney models from four-phase CT studies. The kidney models include the kidney outer contour, kidney arteries, veins, urine collecting system and tumors when present. The segmentation methods rely on the mutual intensity distribution between the CT scan phases, region-growing, and morphological operations. The models are incorporated into a preoperative planning system that allows the surgeon to visualize the kidney and its internal components and to position a resection plane. A validation study on three four-phase CT sequences comparing automatic vs. manually segmented models of the kidney and its internal components yields a mean overlap error 1.7% for the kidney contour, 14.1% for the ureter, and 25.65% for the blood vessels. 3

4 Contents 1. Introduction Kidney Anatomy Four-phase CT acquisition protocol Laparoscopic partial nephrectomy Goals Methods overview Novel aspects Thesis organization Literature review Kidney segmentation Blood vessels segmentation Preoperative planning Method Kidney ROI localization Registration between CT phases Automatic kidney components segmentation Preoperative planning Experimental results Datasets description Qualitative evaluation Conclusion Summary Discussion Future work Bibliography 46 4

5 List of Figures 1. Kidney internal anatomy and its surroundings 7 2. Kidney anatomy 8 3. Slice samples from the 4-phase CT acquisition Methods flow diagram Four stages in the search for kidney ROI Affine registration Mutual histogram Border between Gaussians Segmentation by volume rendering Combined models with separating plane Arteries segmentation, results and comparison to ground truth Veins segmentation, results and comparison to ground truth Ureter and collectint system segmentation, results and comparison to ground truth Kidney segmentation, results and comparison to ground truth Combined components segmentation, results and comparison to ground truth. 42 List of Tables 1. Scans characteristics Comparison results between the segmentation and the ground truth, for each kidney component. 37 5

6 Chapter 1 Introduction Kidney surgery is a common and complex procedure. The procedure is required to remove kidney tumors and to treat a variety of renal malfunctions. Nowadays, a minimally invasive approach is the method of choice for partial nephrectomy operations. This procedure presents many challenges to the surgeon stemming from the kidney anatomy and the nature of the minimal invasive technique. Preoperative planning based on Computed Tomography (CT) scans is a crucial step, since it can reduce the risk of the procedures and decrease their duration. Many of the surgical decisions can be pre-determined in the preoperative planning stage. Today, surgeons perform preoperative planning by browsing through CT slices and constructing a mental representation of the kidney, its internal components and their spatial relations, and planning the surgery accordingly. This approach is time-consuming, incomplete, and inaccurate. This thesis describes semi-automatic methods for the creation of patient-specific kidney models from four-phase CT studies. The kidney models include the kidney outer contour, kidney arteries, veins, urine collecting system and tumors when present. The models are incorporated into a preoperative planning system that allows the surgeon to visualize the kidney and its internal components and to position a resection plane Anatomy background We begin with a brief description of the kidney anatomy and its relative location in the body. The kidneys are a pair of organs lying on either side of the vertebral column between the peritoneum and the back wall of the abdominal cavity at the level of the 12 th thoracic and first three lumbar vertebrae. The right kidney is slightly lower than the left because the liver occupies a large area above the kidney in the right side. Figure 1 shows the kidneys location in the body. 6

7 Fig. 1: The kidney anatomy and its surroundings [2] An adult kidney is about 12 cm long. Near the center of the medial border there is an indentation called the renal bilum, through which the ureter leaves the kidney and blood vessels, lymphatic vessels and nerves enter and exit. Surrounding each kidney is a smooth, transparent renal capsule, a connective tissue sheath that helps maintain the shape of the kidney and serves as a barrier against trauma. The adipose tissue anchors the kidney to the posterior abdominal wall. The kidney has two main regions: the outer renal cortex and the inner renal medulla. Within the renal medulla are several cone-shaped renal pyramids. Extensions of the renal cortex called renal columns fill the spaces between renal pyramids. Figure 2 shows the kidney anatomy. 7

8 Fig. 2: Kidney anatomy Urine formed in the kidney drains into a large, funnel-shaped cavity called the renal pelvis. The rim of the renal pelvis contains cuplike structures called major and minor calyces. Urine flows from several ducts within the kidney into a minor calyx and from there through a major calyx into the renal pelvis, which connects to a ureter. From there, the urine is extracted from the body. Inside each kidney, the renal artery divides into smaller and smaller vessels that eventually deliver blood to the afferent arterioles. Each afferent arteriole divides into the glomerulus - a tangled capillary network. From there the blood continues to bigger and bigger veins, till ultimately it drains into the renal vein [1] Four-Phase CT acquisition protocol Computed Tomography (CT) scans with or without contrast agent are used to image the kidney. X-ray computed tomography is a non-invasive medical imaging method employing tomography created by computer processing. Digital geometry processing is used to generate a three dimensional image of the inside of an object from a large series of twodimensional X-ray scans taken around a single axis of rotation. Since its introduction in the 1970s, CT has become an important tool in medical imaging. To increase the imaging quality of specific soft tissues, Iodine is sometimes injected to the patient as a contrast agent. The contrast agent gives the containing tissue a brighter respond in 8

9 the image. The purpose of the contrast agent is to enhance the kidney components, which have only little difference from the background in standard CT scans [14]. CT is regarded as a moderate-to high-radiation diagnostic technique. It is estimated that 0.4% of current cancers in the United States are due to CTs performed in the past. The contrast agent may also induce kidney damage. The risk of this is increased with patients who have preexisting renal insufficiency [13]. The CT scans are acquired with a four phase protocol which consists of: 1. A basic phase - a standard CT scan without contrast agent; 2. Arterial phase after the patient is injected with contrast agent to enhance the arteries; 3. Venous phase - acquired when the contrast agent is in the veins; 4. Ureter phase or the Collecting System phase - when the contrast agent is in the ureter. Figure 3 shows a representing slice for each phase Laparoscopic partial nephrectomy We now explain the principle of laparoscopy and discuss the challenges it arises. The main potential users of our methods are surgeons that use the minimal invasive surgeries technique a.k.a. laparoscopy in renal surgeries. One such surgery may be partial nephrectomy surgery, which is a resection of a part of the kidney that has a tumor. In laparoscopic surgery, few keyholes are made in the patient body. Through the trocars inserted in those keyholes, 5-10mm diameter instruments can be introduced by the surgeon into the abdomen. In most surgeries, the abdominal space is filled with CO 2 to create a visible workingspace. A small video camera is introduced through one of the trocars, with which the abdominal space is shown on a monitor in the surgery room. The surgeon does the operation looking at the monitor, while controlling through the other trocars the graspers, scissors and clip appliers. Only several operations can be made by minimal invasive methods, including partial nephrectomy. The main advantage of minimal invasive surgery is the reduced damage to other tissues. This implies reduced pain due to smaller incisions and hemorrhaging, and shorter recovery time. The laparoscopic surgery is challenging to the surgeon, since it requires very good hand-eye coordination. The surgeon sees the operated organs on a monitor, in only partial field of view, and limited resolution. This can cause a problem of anatomic recognition. Also, the camera maneuvering is bounded to only several views, so some important details might be hidden behind other anatomical parts or smoke generated by using burning techniques. Important anatomical parts like blood vessels may be hidden inside organs. 9

10 Fig. 3: Slice samples from the 4-phase CT acquisition: representative sagittal slice of a kidney: a. (top left) - Basic phase; b. (top right) - Arterial phase; c. (bottom left) - Venous phase; d. (bottom right) - Collecting System phase. 1.4 Goals Our goal in this thesis is to create a semi-automatic preoperative planning system. Specifically, the goals are: Accuracy: the method should estimate the segmentation of different kidney components with high accuracy. Ease of use: the preoperative planning system user interface should be intuitive to the surgeon. Clinical relevance: the method should be fully implemented on a working prototype so it can be evaluated clinically. This system must show the user clearly each component, and the components relative locations. It also should give the surgeon a way to simulate different surgery alternatives, and compare them. 10

11 Some of the challenges described in Section 1.2 can be addressed with preoperative planning. By doing a preoperative planning the surgeon can know in advanced where to expect each component, from which direction to approach the kidney, and how to perform the resection. A semi automatic preoperative planning system may therefore be useful. 1.5 Methods overview The preoperative planning system consists of two components: Models Generation and User Interface. Model generation- In this stage the input scans are used to generate models of the kidney components. The input is 4 phases CT scans. First, we find the Region Of Interest (ROI) of the kidney in each CT scan. Secondly, we register the ROIs to each other. Thirdly, we segment the internal components using the registered ROIs. The components are the kidney outer surface and its arteries, veins, ureter and tumors. For segmentation we use the mutual distribution method which consists on the distribution of the gray values in the different scans. At last, we combine the models into a single combined 3D color model of the kidney and its components. User interface With a graphical user interface (GUI), the user can see the model of the kidney and its components. He can rotate it to get a better perception of the kidney s components and their relative location. He interactively simulates different surgery courses, and chooses between them, while the system compares the different options. 1.6 Novel aspects This thesis makes five main contributions to the state of the art: 1. Segmentation by mutual distribution: We designed and implemented an algorithm to segment different components from multiple scans using the characteristic mutual distribution of the gray values inside the wanted component, and the mutual distribution of the background. 2. Segmentation by volume rendering back projection: We designed and implemented an algorithm to segment any component that can be distinguished in a volume-rendering projection image, by first segmenting it in the projection image and then re-projecting it to the original 3D image. 3. Kidney localization: we developed and implemented a system to locate a ROI containing the kidney in four-phases full-torso CT scans. 11

12 4. Expectation Maximization initialization: We developed and implemented an initialization process which changes the inputs to a Expectation-Maximization (EM) process, making it solve to the global maximum. 1.7 Thesis Organization This thesis is organized as follows. Chapter 2 presents literature overview about segmentation and preoperative planning. Chapter 3 presents the methods we used. Chapter 4 describes experimental results. Chapter 5 presents the conclusions. 12

13 Chapter 2 Literature review Medical image segmentation is one of the most widely researched fields in the computational biomedical community. Preoperative planning is also a very popular research area. However, to the best of our knowledge, there is no research that combines segmentations of the internal kidney components for a preoperative planning of partial nephrectomies. We survey in this Chapter the previous research that is relevant for the task. Kidney segmentation There are relatively few works on kidney segmentation. Most of the works on kidney segmentation do not assume the present of a contrast agent. El-Baz et al [2] segmented the kidney using an atlas. Ali et al. [3] used the Poisson distribution and distance maps to compute the shape term of the graph for the segmentation of 2D kidney slices from DCE-MRI. Recently Freiman et al [4] described a non-parametric shape information in a min-cut framework. In comparison to the task of kidney segmentation, there is a vast amount of research about segmentation of other organs. The most relevant organ is the liver. The liver segmentation methods can be classified into four main categories: (1) Atlas-based methods: existing segmentations from a training set are used to build an atlas a generic model which is used to find the liver in the new CT scan, and to guide its segmentation. Lamecker et al [21] prepared an atlas manually and compared shapes to each other using principal component analysis. They didn t extend this comparison method to a segmentation algorithm. In Heimann et al [22] 3D Active Shape Models were generated based on the minimum description length (MDL). The search was performed with three different grayvalue appearance models; a deformable model coupled to a shape model was then used to do the actual segmentation. Okada et al [23] combined probabilistic atlas (PA), which was first used to obtain a liver region, and statistical shape model (SSM) to obtain a more accurate segmentation. 13

14 (2) Region growing methods: these methods typically require a seed - an annotation from the user - of a small area which is inside the liver. The region contained in the seed is then enlarged iteratively. In each iteration the segmentation rule is re-estimated, and then is applied again on a close region of the last iteration result. The segmentation rule is a homogeneity criterion, which assumes the segmented part gray values are homogeneous. Pohle et al [24] developed a region growing algorithm that learns its homogeneity criterion automatically from characteristics of the region to be segmented. The method is based on a model that describes homogeneity and simple shape properties of the region. Parameters of the homogeneity criterion were estimated from sample locations in the region. These locations were selected sequentially in a random walk starting at the seed point, and the homogeneity criterion was updated continuously. Nakayama et al [25] selected seeds automatically by comparing patches to a known liver-mean-gray-value. The seeds were than extended using restricted growing regions, using edge information from Sobel filter as a segmentation rule. The results were validated using comparison of the segmented volume to the actual volume of the removed liver. (3) Deformable models and level sets based methods: these methods evolve a contour towards the lowest potential of a cost function that combines gradients and smoothness constraint. The key idea is to represent the evolving contour using a signed function, where its zero level corresponds to the actual contour. The main drawback of these methods is that the convergence of the process is not guaranteed, and requires the user to stop the segmentation process manually. Pan et al [26] developed a level-sets segmentation method with a speed function that is designed to stop the propagating front at organ boundaries with weak edges, and incorporate a-priori information on the relative position of the liver and other structures. Schenk et al [27] designed an interactive segmentation system using shape-based interpolation to approximate the contours on slices between user-defined boundaries. (4) Classification based and graph based methods: these methods address the segmentation problem as a regular voxels classification problem, for which machine learning and classification methods can be used. Sosna et al [29] used Bayesian likelihood maximization as such learning method. Lambrou et al [28] used Quadratic Classifier (QC) to classify tumor and normal liver voxels, when the features to the learning process were computed at different scales of the wavelet transform analysis and from statistical pattern recognition methods. Since these methods treat each voxels separately in the classification process, a global constraint should be combined, using graph-cuts maximization scheme. Our method belongs to this category, since we use the Expectation Maximization machine learning algorithm for classification. Since many of the above mentioned methods may produce poor results, researches have proposed a hybrid two-phase approach consisting of a preliminary segmentation (usually atlasor intensity-based) followed by a deformable model refinement. Gao et al [19] used first a threshold-based segmentation with different threshold for each liver section. They then refined the boundaries by optimization of the parametrically deformable contour model. Soler et al [20] 14

15 segmented the kidney using gray-level histogram analysis of the image, and used two-stage algorithm for the segmentation of the liver: they used first an atlas-based method to get an initial segmentation of the liver, and then used deformable models to refine it. This method works well when the preliminary segmentation is close to the global maxima. If it is off due to local minima, it is very unlikely that the refinement step will manage to significantly improve it. Metaxas et al. [18] propose a hybrid iterative scheme for general object segmentation. In this scheme, the results of one method are used by the second one iteratively until convergence. Specifically, this method first computed a Markov Random Field from manually selected initial pixel seeds inside each organ, and then used it as the initial mesh for deformable model based segmentation. The process was repeated with the resulting contour until convergence. The advantage of this method is that while each iteration produces only small modifications, the accumulated modification can be significant. However, the computation of 3D Markov Random Fields is computationally expensive, so the implementation used 2D fields instead, with the consequent penalty in accuracy. Liver segmentation methods include automatic and semi-automatic methods. The disadvantages of the semi-automatic methods [24 27, 18] are that most of them require significant user interaction, such as the selection of multiple pixel seeds, or the manual drawing of initial contours. This initialization is time-consuming and user-dependent, which significantly affects the robustness of the methods. Because of the liver boundary ambiguity, expert radiological knowledge, which is not always available, is often required to obtain a proper initialization. Automatic methods [21 23] do not require user initialization but depend on good prior shape models, which are sensitive to shape variations between healthy and sick livers, different patient ages, and image scanning protocols [8]. A third alternative is nearly automatic methods, which require little, non-expert user intervention. Freiman et al [8] designed a nearly automatic, hybrid liver segmentation algorithm that uses an iterative classification scheme. The algorithm repeatedly applies multi-resolution smoothed Bayesian classification followed by adaptive morphological operations and active contours refinement. The smoothed Bayesian classification uses patient-specific soft threshold coupled with neighborhood information for more accurate classification. Adjacent organs and tissues are explicitly removed during the classification step using an intensity distribution model developed for this purpose. The resulting segmentation is refined with an active contours step. Blood vessels segmentation Vessels segmentation is of great importance, since every organ in the human body has blood supply. A good review of vessels segmentation can be found in [17]. We can divide blood vessels segmentation algorithms into three main categories: 15

16 (1) Pattern recognition methods: In the vessel extraction domain, pattern recognition techniques are concerned with the automatic detection of vessel structures and the vessel features. Chwialkowski et al [16] performed contour detection approach, using multi-resolution analysis based on wavelet transform. Tozaki et al [15] used a skeleton-based approach which extracts the bronchus and blood vessels from CT scans of the lung. As a first step, a threshold was used to segment the scans. Then, blood vessels and bronchus were differentiated by using their anatomical characteristics. Finally, a 3D thinning algorithm was applied to extract the centerline, which gave the skeleton of the blood vessels. The resulting skeleton was used to analyze and classify the blood vessels. Region growing methods can also be considered as pattern recognition techniques. Schmitt et al [32] combined thresholding with region growing technique to segment vessel tree in 3D. Higgins et al [31] added to the growing regions technique a cavity filling process to add the cavities missed during seeded region growing process. The most popular pattern recognition approach is matching filters. Matching filters approach convolves the image with multiple matched filters for the extraction of objects of interest. Sato et al [30] introduced a 3D multi-scale line enhancement filter for the segmentation of curvilinear structures in medical scans. The 3D line filter was based on the directional second derivatives of smoothed scans using Gaussian kernel using multi scales with adaptive orientation selection using the Hessian matrix. (2) Model-based methods: In model-based methods, explicit vessel models are used to extract the vasculature. This can include wide spectrum of models, from deformable models, or active contour to template matching. Kass et al [33] were the first to use active contour models or snakes. Hu et al [34] present a method based on global and local deformable physical models to extract vessel boundaries. The method uses a circular global model which fits the shape of the vessel cross-section boundary. The local model, with variable stiffness parameters, locates the contour on the edge point locations where edge features are strong while keeping the contour smooth at the locations where edges are missing. Edge segments are extracted using directional gradient information. Level sets methods can also be classified as model based approach. Caselles et al [35] and Malladi et al [36] use propagating interfaces under a curvature dependent speed function to model anatomical shapes using level sets method. Sethian [37] developed a similar method, called the Fast Marching method, which uses a wave propagation approach for specialized front problems. Krissian et al [39] developed a multi-scale model: first, they created a skeleton from local-maximum image, and then extended the skeleton to a full segmentation by using a new response function which measures the contours of the vessels around the centerlines. In template matching for arterial extraction applications, the arterial tree template is usually represented as a series of connected nodes in segments. This template is then deformed to optimally fit the structures in the scene. A simpler parametric template matching is the Generalized Cylinder Model. Kayikcioglu and Mitra [40] used a parametric model with elliptical cross-sections to reconstruct coronary arterial trees from biplane angiograms. 16

17 (3) Tracking-based methods: Tracking-based methods apply local operators on a region containing a vessel and track it. Starting from an initial point, they detect vessel centerlines or boundaries by analyzing the pixels orthogonal to the tracking direction. Aylward et al [41] applied intensity ridges to approximate the medial axes of tubular objects such as vessels. A more sophisticated approach on vessel tracking is the use of graph representation [42]. The segmentation process is reduced to finding the optimum path in a graph representation of the image. Freiman et al [38] described a graph-based segmentation method for modeling of the aortic arch and carotid arteries from CTA scans. The method starts with morphological-based segmentation of the aorta and the construction of a prior intensity probability distribution function for arteries. The carotid arteries are then segmented with a graph min-cut method based on a new edge weights function that adaptively couples the voxel intensity, the intensity prior, and geometric vesselness shape prior. As for renal vessels segmentation, we did not find any references. On the other hand, there is much work on segmentation of blood vessels in the liver, which has some similarities to the kidney. Charnoz et al [5] used a simple intensity distribution based technique. Groher et al [6] performed a growing-region with a seed provided by the user. Peitgen et al [7] designed an edgebased segmentation technique with multiple seeds. Freiman et al [8] used iterative multiresolution multi-class smoothed Bayesian classification with active contour refinement. Recently, Alhonnoro et al [9] published a work which is the most similar to our thesis: they use 3-phases CT scans for segmentation of blood vessels using multi-scale vessel enhancement filter, and then present preoperative planning for liver RFA procedures. Preoperative planning There are preoperative planning systems for many kinds of surgeries. Most of them deal with inserting a needle into a tumor, and then eliminating it with different methods such as microwave, radio frequency (RFA), thermal ablation or cryoablation. Again, liver is the most similar organ to which there are preoperative systems. Baegert et al [10], and recently Alhonnoro et al [9] designed such a system for RFA therapy and Zhai et al [11] designed one for microwave ablation. In the context of brain surgeries, Joskowicz et al [43] designed a preoperative planning system for choosing a trajectory to a brain tumor, maximizing the margins between the trajectory and important structures like blood vessels. For partial nephrectomy, Ukimura et al [12] presented 3D surgical model, named color-coded zonal navigation which gives a different color for different margins from the tumor. This model can be interpreted as a preoperative planning, even though it does not take into consideration other kidney components like blood vessels, but leaves it to the surgeon. 17

18 To summarize the literature review, we conclude that even though abdominal organs segmentation, blood vessels segmentation and preoperative planning are all widely discussed topics in the medical image processing community, kidney and kidney components segmentation for minimal invasive partial nephrectomy preoperative planning has not yet been addressed. 18

19 Chapter 3 Methods This section discusses the methods for kidney surgery preoperative planning. We start with a system overview and then describe each component in detail. Our preoperative planning system consists of kidney inner segmentation and a user interface. Our kidney segmentation and modeling method from 4-phase CT scans consists of four steps (Fig 4): The first stage is finding kidney ROI in the full torus CT scans. The second stage is registration of the CT scans based on the kidneys ROI. The third stage is automatic kidney inner segmentation the kidney itself and each of its components are segmented and combined into a single model. The fourth stage is preoperative planning a user interface for using the combined model to plan the surgery. 3.1 Kidney ROI localization We find the location of the wanted kidney in each scan, by finding a cubic, axis-parallel, Region Of Interest (ROI) which bounds the entire wanted kidney. This is performed in two stages: (1) locating kidney in the artery phase, and; (2) using it for locating the kidney in the other three phases. Locating kidney in the arterial phase In the arterial phase, the contrast agent reaches the interlobular arteries, and therefore most of the kidney is marked as brighter in the CT scan. Therefore, finding the kidney in the arterial phase is easier than in the basic phase. 19

20 Fig. 4: Flow diagram for the methods section We scan the different possible gray-scale windows; each window has 10 gray-level; each of yields a 3D segmentation mask on the original scan voxels which are in the gray-level window are white, and the others are black. The areas with contrast agent have representative gray level value. The goal is to find this typical value, and thus to find the kidney ROI. We use volume rendering, which in the case of a binary mask is a simple maximum operator on each column of the 3D mask in the coronal direction. This gives us a 2D max-image for each of the possible gray level windows. We did this because we used very narrow gray level windows, 20

21 (a) (b) (c) (d) Fig. 5: Four stages in the search for kidney ROI. Each couple is a projection of the two leading connected components in the binary segmentation decided by a certain gray values window. (a) gray values window of (b) (c) and (d) By using the symmetry distance described in section 3.2 we can choose (b) as describing the and the 3D mask might be too sparse to process as is. By finding the one 2D mask which best suites the kidney, we will find the right gray scale window. Finding the 2D-mask is done by symmetry distance. In each 2D max-image, we identify the two largest connected components, and their centers. One symmetry distance is the euclidean distance between the centers of the components after mirroring one of them. The second symmetry distance is the percent of difference between the two shapes, after one of them was mirrored, and co-registered using their centers. The actual distance is normalized multiplication of those two. This is a distance because lower grade means the two components are more symmetric to each other, and if the two components are exact mirror of each other, they will get zero distance. This is not a geometrical distance, because the distance of an image to itself is not 0. Since the two kidneys are more or less symmetric, both in their relative location and in their shape and they are the only two symmetric abdominal organs, finding two symmetric organs in an image will indicate that this is an image of kidneys. Once we have chosen a 2D max-image, which has kidneys masks in it, we apply it to the 3D segmentation mask, which it was created from, and find one main connected component in it. At this stage we choose the wanted kidney, left or right, by looking at the relevant half of the CT scan. The bounding block of this component will be the ROI containing the kidney. Locating the kidney in the remaining three phases We use the result of the previous step to find the kidney ROI in the other phases. First, we reduce the search area by anatomy knowledge; in the arterial phase the kidney ROI is in a certain location, it will appear in other phases in the transversal direction up to 10 cm. But in the other two directions it may move only 2 cm. This constrains the search area, saves us computation time and avoids false matches. 21

22 We use for the actual search Normalized Cross Correlation (NCC) as a similarity grade. The kidney ROI from the arterial phase is used as a template which we will search in the other phases. The slice by slice search is done in 2D. For each of the possible three directions (sagittal, coronal and transversal) we compute a similarity grades map. In this similarity grades map, each slice is the NCC grades map between the corresponding slice in the search area and the middle slice of the kidney template. To get a unified 3D similarity map, we multiply the three 3D similarity maps. The location of the maximal grade in this 3D similarity map will be the center of the result ROI. The ROI size is the same as in the arterial phase. The scans are shrinked in all directions by a factor of 2 to reduce memory and running time in a factor of 8, without compromising on accuracy. The search is performed by 2D NCC because 3D NCC is a very time-consuming process. The NCC similarity grade was chosen because of its relative speed, and its ability to handle the changes in brightness between the scans. 3.2 Registration between phases We now present the registration between the kidneys ROI in the different phases. We apply affine 3D transformation as a registration model. The search for the right affine registration parameters is made in a gradient-descent method, when Mutual Information (MI) is used as a grade to guide the search. The registration is performed using the SPM package, and tight ROIs around the kidney were used. The arterial phase was chosen as fixed, and the other three ROIs were registered to it. Figure 6 illustrates the registration process. The kidney is a deformable organ. During the breading cycle, two main forces operate on the kidney: the adipose tissue anchors the kidney to the posterior abdominal wall, while the renal artery prevents it from distancing from the aorta-renal artery junction. These forces cause the kidney to move along the transversal direction, with a small rotation. It also causes a small nonrigid move. Because of that we chose affine 3D transformation as a registration model, which can model the big shift and the rotation, while it approximates the non rigid move. Choosing tight ROIs around the kidney prevented the big non rigid move of the organs which are outside of the kidney. The flow of the contrast agent through the kidney in the different phases changes the gray level of the kidney components. This change is not monotone. The ureter might be brighter than the arteries in the collecting-system phase and darker than it in the arterial phase. This rules out gray-level based grades such as NCC, SSD and SAD. It also rules out popular registration algorithms that are gray-level based such as Mellin-Fourier transforms and optical-flow 22

23 Fig. 6: Affine registration. Upper row: The mutual histogram used to calculate the mutual information grade, which is maximized in the registration process. Presented the histogram before (a) and after (b) registration. Lower row: the registration results. (c): the fixed image, a kidney ROI from the arterial phase, three central slices. (d): the moving image, a kidney ROI from the collecting system phase, registered to the fixed image, three central slices. 23

24 algorithms. Regular gray-level normalization, used in the NCC, is not applicable since the change is not monotonic. We thus chose Mutual Information as the registration grade, to incorporate the non monotonic changes. We choose the arterial phase as fixed since it is used in the arteries segmentation, which are the smallest component we segment, therefore the re-sampling process during the registration may have a big affect on this segmentation. 3.3 Automatic kidney inner segmentation This section discusses the segmentation of the kidney components: the arteries, the veins, the ureter, and the tumors when present. First we explain our main segmentation method - segmentation by mutual distribution. The segmentation of each component is then discussed individually: segmentation of bones and aorta, and two alternatives for arteries segmentation. Segmentation by mutual distribution Once the registration of the scans is obtained, each voxel has four values corresponding to its gray level in the four different phases. We model each voxel with a 4D vector, distributed as a mixture of 4D Gaussians. We also assume that each Gaussian represents a single kidney component which has been identified. The segmentation by mutual distribution is a generalization of the angiogram protocol. In an angiogram, an X-ray image of the vessels is taken before and after injection of contrast-medium, the first image is then subtracted from the last to get scans of the vessels. As a generalization of that, the segmentation by mutual distribution replaces the subtraction with a general function which takes the 4D vector values and returns a binary answer a segmented model. We model the data as a mixture of Gaussian distributions. The first task is to find the parameters that define each Gaussian its mean and variance. The parameter values are computed by Expectation-Maximization (EM) process. The first step is computing a starting point for the initialization of the EM process. First we create the 2D histogram for the two scans. We apply a smoothing filter to it and extract its peaks with maximum filter. We use the resulting value as an approximation of the mean value since the maximal value of a Gaussian is its mean value. We use a predefined constant for the variance. Figure 7 shows the 2D histogram of the basic phase and arterial phase scans. From the parameterization of the mutual distribution we create a segmentation thresholding rule, in which given the gray values of a voxel in both scans, we can decide if it belongs to the segmented component or not. 24

25 To create the segmentation rule, we first identify which Gaussian belongs to the segmented component. This is done by specific features of the desired components and the CT phases. Next, we use the resulting Gaussian to determine four thresholds, the minimum and the maximum threshold for each phase. When a voxels gray value is greater than the minimum value and less than the maximum value in both phases, it is associated with the segmented component. Those four values are each computed as the border between two Gaussians. The border between two Gaussians is: A, B ~ N( µ, σ ) µ A µ B Threshold = µ A + σ A ( ) σ + σ where A,B are two neighboring Gaussians, µ A and µ B are the mean value of A and B respectively, and σ A and σ B are the standard deviation values of the A and B respectively, We apply the resulting classification rule to each voxel in the ROI, to obtain the initial segmentation. Using different topological operations for closing holes and removing noise, the final segmentation is created. Using the open source tool ITKSnap, we create a mesh-based 3D model for visualization of the segmentation. Figure 8 shows an example of mixture of Gaussians and the border between Gaussians. Segmentation of the kidney boundary We segment the kidney outer contour by using the method of segmentation by mutual information. The CT phases are the arterial and veins phases. Ribs segmentation The ribs which are next to the kidney are included in most of the ROIs. To segment them, we used the 1D distribution, and segment them using the highest-mean Gaussian. The ribs are bones, so they appear with high contrast in the CT image. We use the one-dimensional distribution for this segmentation because the kidney moves in the breathing cycle relative to the ribs, so the affine registration (section 3.3) does not apply to the ribs. A B 25

26 Fig. 7: Mutual histogram: 2D histogram of the gray values of two co-registered images. The first axis is a kidney ROI from the basic phase; the second axis from the arterial phase. The histogram shape shows that the mutual distribution is a mixture of Gaussians. Veins segmentation The Gaussians that characterizes the veins distribution coincides with the kidney Gaussian. Thus, their means are very close to each other, so the EM process combines them to one Gaussian. To solve this problem we selected only voxels that are between σ to 2σ where σ is the standard deviation of the combined Gaussian. This yields sparse segmentation within the veins. A topological operation of dilation is used to connect the disconnected components, followed by erosion to remove noise. 26

27 Fig 8: Border between Gaussians: A mixture of two Gaussians with partial overlap. The border between the Gaussians (dashed line) is computed with equation (1) and used as a threshold for classifying voxels of the different components. Locating and segmenting the aorta The next step is to segment the aorta. The ROI containing part of the aorta is determined by anatomical knowledge: the aorta is between the two kidneys in the transversal and sagittal direction, and between the middle of the body and the back of it in the coronal direction. We use the Hough transform on each transversal slice in this phase. We search for circles with a diameter of about 20 pixels. The aorta is a large tubular structure which is much brighter than its surroundings in the arterial phase. We compare using Normalized Cross Correlation (NCC) between the result of the Hough transform and a template of a bright full circle with a diameter of about 20 pixels. We threshold the result, and then we chose the biggest connected component. Aorta segmentation yields the mean µ and the variance σ 2 of its gray values, so we select voxels whose gray value is one σ away from µ. We then select the largest connected component. These two operations yield, in addition to the aorta, the two renal arteries. 27

28 Arteries segmentation: solution 1 mutual distribution To segment the arteries, we have developed two methods. The first method performs segmentation by mutual distribution with the distributions that are learned from the aorta segmentation. The reason is that the renal arteries occupy only a small percent of the overall amount of voxels in the kidney ROI, so the relevant Gaussian has very small representation in the 2D histogram. The aorta, on the other hand, is a large blood vessel, whose distribution is readily computed. After segmentation of the aorta, we build the mutual distribution - the mean µ and the variance σ of its voxels. Then we divide the kidney ROI to three groups: voxel whose gray value is inside, above and below one standard deviation σ from the mean µ. Each group is modeled as a separate Gaussian. The boundary between these Gaussians is computed, and a classification rule is made as described above. This classification rule is applies to all of the voxels, to create a segmentation of the middle Gaussian. The final stage is noise reduction. We use the kidney outer boundary segmentation to remove voxels outside the kidney or near its boundary, as the renal arteries become very small near the kidney boundaries, below the actual CT resolution. Arteries segmentation: solution 2 volume rendering back projection We have developed a second segmentation method for arteries when the aorta is not present in the CT. The input is the kidney ROI. In this method, we segment the arteries in the volume-rendered image where they appear very clearly, and then back-project them to the 3D image. First, we create a volume rendering (VR) image of the arterial phase by taking the maximum value from each column in the coronal direction. Secondly, we find maximum for each column in the VR image in the transversal direction. We do not maximize in the sagittal direction, because we don t want the ribs to hide the arteries. This gives us a 1D array of maximal values; some of them are originated from the ribs, some from the arteries, and a minority from the less bright background. We use 1D histogram to approximate the distribution, which is modeled as a mixture of two Gaussians the ribs and the arteries. The maximal points in this 1D histogram approximate the means. EM process refines it and finds also the variances of the two Gaussians. The Gaussian with the lower mean is assigned to the arteries, with mean µ and variance σ 2. The VR image is then segmented using these Gaussian parameters. Pixels less than 2σ away from the mean are labeled as belonging to an artery. 28

29 (a) (b) (c) (d) Fig 9: Segmentation by volume rendering: main steps of the process: a) a central slice of a kidney ROI from the arterial phase; b) a volume rendering image obtained from the same image ROI. The arteries appear in light gray; c) segmentation of the arteries from the volume rendered image; d) final result after back projection to the 3D space and region growing. We generate VR images from 12 different views and segment each of them using the Gaussian parameters computed from the first VR image. We use several views to identify the arteries to overcome possible occlusions by other structures. The resulting 2D segmentations are then back-projected to the 3D domain using arg-max: for each pixel that was classified as an artery pixel, we find the voxel it came from in the 3D image. This voxel is marked as true in a binary 3D mask in each of the 12 views. The resulting segmentations are then combined into a single classification. Next we perform an iterative region-growing process. We use the arterial phase and the venous phase. We apply the Sato filter on the two scans, go get tubularity images. The process is done using these four co-registered images. At every iteration, we compute a 4D Gaussian from the voxels which are already considered as arteries and the values they have in the four scans. The Gaussian mean µ and variance σ 2 is calculated. A neighboring environment of the current segmentation is calculated, and each voxel in it is classified; if it is within one standard deviation σ from the mean µ, it is added to the segmentation. This iterative process extends the results of last stage. This iterative process is stopped after five iterations to avoid leaks and divergence. Finally, we select the largest connected component as the segmentation result. Figure 9 illustrates with an example the main stages of this segmentation method. Tumors segmentation The tumors segmentation is initialized with a user-defined voxel seed, one for each tumor. We then perform a region-growing process around each seed as described in section on the arterial and venous phase scans. The seed is required to be in the center of the tumor. The results from the different annotations are then combined into a single segmentation mask. 29

30 Ureter segmentation For the segmentation of the ureter and the collecting system, we performed segmentation by mutual distribution on basic phase and the collecting system phase. Models All of the above segmentations result in binary masks, which are co-registered since the original scans were co-registered to begin with. We combine the masks into a single mask in which each component has a value, from 1 to 5. We use ITKSnap to create color models from it. For intuitive interpretation of the models, we chose colors that best represent the components: the arteries are red, the veins blue, the ureter yellow, the tumors green and the kidney itself half transparent brown, so the inner components can be seen through it. Fig. 10 shows an example. 3.4 Preoperative planning In this section we will explain how the inner segmentation achieved in the last sections can help us creating a tool to be used by the surgeon in the preoperative stage. First we will explain about the separating plane, and how can we initialize it. Then we will give some options for the plane presentation to the user, and for how the user can control it. Separating plane as a simple tool for planning In partial nephrectomy the surgeon separates between the tumor and the rest of the kidney, while avoiding important components - the ureter and the blood vessels. The simplest way to imagine the cut is with a plane that will separate between the two the whole tumor will be on one side and the important components will be on the other side. The separating plane provides the required information to the surgeon. One of those questions might be from which direction the surgeon should approach the kidney, and which parts of it must be exposed. Other questions might be how dangerous this surgery is going to be and what is the confidence interval between the cutting plane and important components. Using the separating plane, we can calculate precise quantities to different parameters that might be of the interest to the surgeon, like the volume of the remaining kidney tissue. Other separation models are possible: some surgeons use a curved manifold, some use two or more planes, and some use the manifold defined by a certain margin from the tumor [12]. We won t discuss these options in this thesis, because designing a graphic user interface for them will be quite cumbersome. 30

31 Finding initial plane We first find an initial plane, which the user can then adjust as needed. This cannot be the final separating plane, because there might be considerations that cannot be predicted in advance. We can give the user the best separating plane which takes into account all of the known considerations, so the user will have a good starting point. For this initial plane, we compute the center of mass for the ureter voxels. Then we find the tumor voxel that is the closest to this point. The plane will be perpendicular to the line connecting those two points, and the resulting degree of freedom is the distance of the plane from the ureter center. For each possible distance, we calculate how much of the ureter or the tumor is intersecting with the plane. If there is indeed a separating plane defined by those two points, than there should be a segment on the line connecting the two points, that every point on it defines a good separating plane. For maximizing the margins of the plane, we choose the middle of this segment for defining our plane. Display Once we have an initial plane, we present it to the user. One simple option is to create a model of the plane, one voxel wide, and to show only the part of it which is inside the kidney. The disadvantage of this representation is that from almost any view angle, the plane hides something. So for each possible plane, the user should manipulate the combined model with the separating plane, so he might see it from different angles, in order to understand how the plane is placed in reference to other kidney components. Another option is to virtually cut the kidney with the separating plane, and to show the two resulting parts separately. This provides a good intuition for what exactly the plane does. Currently, we have implemented only the first presentation option. The disadvantage is that it is very uncomfortable to manipulate the plane when it is not visible. However, this is a good representation for the last stage, when the user has decided on one or more preferred separating planes. User interface The interface between the user and the preoperative planning system consists of two stages: planning and verification. In the planning stage, the user changes the orientation of the plane relative to the models. The user must be able to move and rotate the plane, while the models stay still. When the user has chosen the desired orientation, the verification starts. In this stage, the selected plane is fixed to the models, and the user rotates them together so he can see from different view angles how the plane is oriented. In this second stage, we can also give the user the virtual cutting presentation option described in section The user can always return to the planning stage and refine his decision. 31

32 Fig. 10: Combined models with separating plane. The different models are combined into a single model. Each component is assigned a color: arteries red, veins blue, ureter yellow, tumor green, separating plane cyan and the kidney half transparent brown. The main difference between the two stages is the number of degrees of freedom (DOF) which the user controls. Both stages have three DOFs: in the planning stage the plane can be defined by three parameters (intersection with the three axes). In the verification stage the user can rotate and zoom on the models, which can be defined by two viewing angles and one scale factor. The challenge comes when the user control six parameters in the planning stage. We consider three options for the user interface for the planning stage: The first option is very common in computer graphics: rotating the plane by dragging the mouse while holding the right mouse button. Moving the plane is done by dragging the mouse while holding the mouse wheel pressed. In the verifying stage, rotating the models with the fixed plane is done by dragging the mouse while holding the right-click pressed. This option is the preferred 32

33 one, but we have to check first how medical users, who do not usually interact with computer graphics system, react to them. The second option uses the keyboard and the scan browser. First, the user annotates a 3D point in the scan browser. This operation is very familiar to medical scans users. This 3D point is used as a pivot the plane will pass through it, and the two remaining DOFs are the plane orientation. They will be controlled by the arrow keys in the keyboard: the up-down keys will control the smallest angle between the plane and the Y axis, and the left-right keys will control the smallest angle between the plane and the X axis. The user can at any stage change the pivot, holding the last chosen angles. The third option uses a pen-like arm 3D mouse such as the Omni phantom. By moving and rotating it to any desired orientation, the system changes the plane orientation directly. This gives a very intuitive user interface. Switching between the planning and verifying stages is performed by clicking on the 3D mouse button. The disadvantage of the 3D mouse is that it is not readily available. 33

34 Chapter 4 Experimental results This chapter presents the validation study of our algorithms on three clinical datasets. We first describe the datasets and then present the results of our algorithm. 4.1 Datasets characteristics We used anonymized datasets of three patients. The patient aliases are: DM, DW and ET. Each of the three databases consists of four phases of CT scans: basic phase, arterial phase, venous phase and collecting system phase, as described in Section 1.2. The scans characteristics are shown in Table 1. The patients were potential kidney donors, therefore no tumors or other abnormalities were present. We used only the left kidney from each of the patients. 4.2 Qualitative evaluation In this section we present the results of the segmentation of kidney components. We implemented our method using the MATLAB software. All the computations were performed on an Intel Core2 Quad 2.4 GHz PC with 4GB of memory. We compared our results to manual segmentation using the ITKsnap GUI. We refer to this segmentation as the Ground Truth segmentation. We present the results of each component separately. 34

35 Table 1: Scans dimensions and resolutions. Scans Resolution s Basic phase Arterial phase Venous phase Ureter phase Scans Dimension s Basic phase Arterial phase Venous phase Ureter phase Sagittal (mm/pixel) ET DM DW Corona l (mm/pixel) Spacin g (mm) Sagittal (mm/pixel) Corona l (mm/pixel) Spacin g (mm) Sagittal (mm/pixel) Corona l (mm/pixel) Spacin g (mm) Slice width (pixels ) ET DM DW Slice height (pixels) # Slices Slice width (pixels ) Slice height (pixels) # Slices Slice width (pixels ) Slice height (pixels) # Slices We compared our segmentation results to the ground-truth using five metrics: 1) volumetric overlap; 2) relative absolute volume difference; 3) average symmetric absolute surface distance; 4) symmetric RMS surface distance, and; 5) maximum symmetric absolute surface distance. Table 2 summarizes the results. Figures show the results graphically for each component. We have chosen the most different slices between the results and the ground truth to illustrate a maximum error estimation. As a qualitative measurement, our results (but not the ground truth) were examined and approved by a radiologist, an urologist, and two minimal invasive surgeons. 35

36 36

37 Table 2. Results of the comparison between the segmentation and the ground truth, for each kidney component. Arteries Overlap Volume Avg. Dist. RMS Dist. Max. Dist. Error. [%] Diff. [%] [mm] [mm] [mm] ET DW DM Average Veins Overlap Volume Avg. Dist. RMS Dist. Max. Dist. Error. [%] Diff. [%] [mm] [mm] [mm] ET DW DM Average Collecting Overlap Volume Avg. Dist. RMS Dist. Max. Dist. system Error. [%] Diff. [%] [mm] [mm] [mm] ET DW DM Average Kidney Overlap Volume Avg. Dist. RMS Dist. Max. Dist. border Error. [%] Diff. [%] [mm] [mm] [mm] ET DW DM Average

38 Arteries ET Sagittal Coronal Transversal 3D model Ground truth Results DM Sagittal Coronal Transversal 3D model Ground truth Results D W Sagittal Coronal Transversal 3D model Ground truth Results Fig. 11: Arteries segmentation, results and comparison to ground truth. 38

39 Veins DM Sagittal Coronal Transversal 3D model Ground truth Results D W Sagittal Coronal Transversal 3D model Ground truth Results ET Sagittal Coronal Transversal 3D model Ground truth Results Fig. 12: Veins segmentation, results and comparison to ground truth. 39

40 Ureter and Collecting system DM Sagittal Coronal Transversal 3D model Ground truth Results D W Sagittal Coronal Transversal 3D model Ground truth Results ET Sagittal Coronal Transversal 3D model Ground truth Results Fig. 13: Ureter and collectint system segmentation, results and comparison to ground truth. 40

41 Kidney DM Sagittal Coronal Transversal 3D model Ground truth Results D W Sagittal Coronal Transversal 3D model Ground truth Results ET Sagittal Coronal Transversal 3D model Ground truth Results Fig. 14: Kidney segmentation, results and comparison to ground truth. 41

42 Combined models DM Sagittal Coronal Transversal 3D model Ground truth Results D W Sagittal Coronal Transversal 3D model Ground truth Results ET Sagittal Coronal Transversal 3D model Ground truth Results Fig. 15: Combined components segmentation, results and comparison to ground truth. 42