2011 Segmentation of Airway in 4D Lung CT Images

Transcription

1 I

2 Acknowledgements I would like to thank my supervisor Derek Magee for his support, motivation, and expert feedback in frequent weekly meetings as well as my assessor Dr Kia C. Ng for his feed back in interim report and progress meeting. Finally I would like to thanks my family for their financial and moral support. II

3 Abstract Lung cancer is a disease that consists of uncontrolled cell growth in tissues of the lungs. There are a number of different treatment options available for lung cancer. Radiation therapy is a process in which high energy radiation is applied to kill cancer cells and shrink tumors. This process also affects the normal cells but they are able to repair the damage. A treatment plan of patient is developed before radiation therapy in which the clinician spent a lot of time on manually marking the airways on the CT scan images so that airways can be protected while doing the radiotherapy. This project looks into automating the airway segmentation. First I have tried to evaluate the existing method and evaluation of existing method was done. Existing method used dynamic programming to fit a model airway on a minimal spanning tree which was generated by the segmentation and Skeltonization of input data. This method doesn t provide the desired results. In 2 nd prototype Point distribution method was used to extract the airway from the minimum spanning tree. Shape models were created by manual annotation of available data and principal component analysis was applied on the shape models, to find out the useful principal components. Manual hypothesis were generated from the minimal spanning tree to evaluate the PDM. RMSE was used as fitness function which calculates the Euclidian distance of shape models to the target points. RMSE values decide whether the hypothesis is correct or not. III

4 Table of Contents Chapter 1: Introduction Project Context: Aim Objectives Minimum Requirements List of deliverables Resources Required Relevance Methodology Project Plan Key Activates Performed Milestones... 8 Chapter 2: Background Literature Review What is Segmentation? Image Segmentation Methods: Data Pre-Processing Methods: Minimum Spanning tree Registration of point sets Previous related work Vessel Segmentation: Airway Segmentation: Chapter 3: Design Overview Data Analysis: Data Preparation: Generation of a Tree model from image data: What is a model? Generation of Hypothesis Manual annotation of points: Aligning the Training model IV

5 3.9 Prototype 1: Dynamic programming Prototype 2: Point Distribution Model Chapter 4: Implementation Understanding Datasets: Aligning Training Models: Generation of the Minimal spanning Tree Prototype 1: Dynamic Programming Prototype 2: Point Distribution Method Iteration 1: The 4-Point Model Iteration 2: The 10-Point Model Iteration 3: The 9-Point Model Chapter 5: Evaluation Prototype 1: Dynamic Programming Prototype 2: Point Distribution Method: The 4-Point model: The 9-Point model: Chapter 6: Conclusion and Future work Conclusion Future Work Improved annotations Increasing the training data Search Strategy to find possible hypothesis Airway contouring and evaluation References: Appendix A: Personal Reflection Appendix B: Resources Appendix C: Minimal spanning tree of datasets Appendix D: Interim Report V

6 1

7 Chapter 1: Introduction 1.1 Project Context: Lung cancer is a disease that consists of uncontrolled cell growth in tissues of the lungs. There are a number of different treatment options available for lung cancer. Standard treatment option includes surgical resection, chemotherapy and radiation therapy [1]. Radiation therapy is a process in which high energy radiation is applied to kill cancer cells and shrink tumors. These high energy radiations damage the DNA in cells which cause them to die. Cancer cells divide more frequently than normal cells so they are more susceptible to damage. This process also affects the normal cells but they are able to repair the damage. A treatment plan of patient is developed before radiation therapy and this process is called as treatment planning [1]. Lungs exchange air with the external environment via the pulmonary airways. This project revolves around the airways, so it is important to understand the structure of an airway. The airways are organized as a branching networks of tubes that become narrower, shorter and more numerous as they penetrate deeper into the lung. Air enters thorough the mouth and nose via pharynx and is conducted to lungs through the trachea. The trachea divides into two main branches, the left and right bronchi, and these branches finally enter into the lungs. The geometry of the airway tree is approximately bipodial, i.e. at each branch division the airway branches into 2 child airways [1]. The branches from trachea to the end of terminal bronchioles make up the first 17 generations of the airway tree. The airway continues to branch and terminate at the alveolar sacs, the site of gas exchange. In treatment planning, clinicians/doctors highlight the airways on CT images to avoid the direct dosage of radiation on the airways, because airways are very sensitive to damage. 12 phases of the breathing cycles are obtained during a CT scan of a patient. Of these phases, clinicians/doctors will manually mark the airways on one phase because it is a very time consuming. In fact, it takes up to twenty minutes to mark just one phase. It would be much more helpful, if all the phases could be marked. A system is required here which could automatically or semi-automatically mark the airways in all the phases of a CT scan. 2

8 Figure 1.1: Structure of airway We can see in Figure 1.1 that the structure of airway becomes very complex after a few generations. After consultation with clinicians in St James hospital, we came to know that, they are only concerned with the early generations while segmenting the airways in treatment planning. After a few generations, the airways become very small and numerous and they become incomprehensible on the CT scan. The clinicians will only mark the part of airways which could be damaged by the radiation. The other parts of the airways are not critical for radiation therapy. Figure 1.2 shows the areas of interest for surgeons, which should be marked in treatment planning. Figure 1.2: Surgeon s Area of interest in Airway 3

9 1.2 Aim The Aim of this project is to help clinicians to save the time they spend on manual segmentation of airways in CT images. Clinicians currently only do the segmentation for 1 phase out of 12 phases for a patient but using this system will aim to segment the airway in most phases. If we can segment the airways in all phases, this would help in quantifying motion which will potentially reduce the treatment margin. This project will also try to provide an improved, useable interface to view/analyze the outcomes after segmentations. 1.3 Objectives The objectives of the project are: Extract airway information from a DICOM dataset. Skeltonization and finally creating a tree model. Creating a training model by manually selecting points on the image. Evaluate the existing methods of segmentation of the airways. Creating a pre-determined model for fitting the airway information. Creating GUI for better ease of use of the Application. 1.4 Minimum Requirements 1) Method for segmentation of airways using a model 2) Method for quantifying motion and/or visualization of airways. 3) Investigation the best way to presenting the results. 1.5 List of deliverables 1) An application which will do the segmentation of Airways and Quantify motion. 2) A method for better visualization of output. 3) A Final report. 4

10 1.6 Resources Required 1) Datasets: There are 6 Different datasets currently available. Each dataset contains 12 CT images of the same patient. While doing a CT scan of a patient, 12 breathing phases of a patient s breathing cycle are recorded so each image dataset represents a breathing cycle of the same patient. 2) ITK: Insight Segmentation and Registration Toolkit (ITK) is an open source system that provides software tools for image analysis. My project will make extensive use of ITK and various other libraries for segmentation and Skeltonization of images. 3) Sante DICOM Viewer: Sante DICOM Viewer is advanced medical software for reading the DICOM files. In the sample datasets, Clinicians have only marked one of the phases and I used this tool to extract that specific phase from the whole dataset. 4) CERR: CERR is a tool which can be used to import and display treatment plans. In my project, I have been using CERR to visualize the marked phase of the dataset to gain an understanding of how doctors mark the airways. CERR cannot visualize the whole dataset, so Sante DICOM Viewer was used to extract that specific phase from dataset. 5) Matlab: Matlab was extensively used for visualization purpose as well as for programming a few parts of the project, such as aligning the shape models using Procrustes analysis. 6) Volume Viewer: A program developed by Derek Magee was used for Viewing Volumes of the available data. 1.7 Relevance This project builds on skills and knowledge gained from a module Vision (COMP5430M). I have been using a lot of methods which were taught in that module i.e. Segmentation, Thinning, Connected component analysis, Point distribution Model etc. Computational Modeling (COMP5320M) helped me to learn how to use Matlab. Matlab has been used in this project for plotting the shape models and trees generated after the Segmentation of dataset. 5

11 1.8 Methodology a) Understanding the problem: I have read the related literature and previous work done in the same domain. b) Searching for possible solutions: Various model fitting methods were available. I tried to find the solution using 2 different approaches. First extension involved simple model fitting or complex model fitting. By that approach I was not able to get the satisfactory results so I tried the second model based approach which was Point Distribution model. c) Implementation: The first prototype was implemented using ITK. Volume Viewer and Matlab were used for the visualization of results. The second prototype was implemented by partly using few existing C++ libraries and partly using Matlab. Matlab was used for the visualization of the results. d) Validating results: After implementation of every extension, Evaluation of that extension was done to check whether that method is feasible or not. e) Report the results: In the last phase of the project a report has been written which shows all the work that has been done. 6

12 1.9 Project Plan Key Activates Performed Figure 1.3: Revised Project Plan a) Research: i) Literature Review: This includes citation of previous work done in the same domain and literature review. ii) Understanding Tools: Understanding of various tools was important to utilize them in a better way. iii) Analyzing Datasets: To search a solution, understanding of CT data was very important. It was very difficult to understand the technical details especially when splits in airway happen. Volume Viewer was mostly used to visualize the datasets to understand various phases of datasets. iv) Preparing Datasets: Manual annotation was done to get training data. This manual annotation was used to make a tree model of airways and it was used in both prototypes of project. v) Design prototypes: Some time was spent on the designing of each prototype before implementation phase. b) Development: Development of this project consists of 2 prototypes. The first prototype was the evaluation of existing methods by making some changes. The second prototype was Point distribution model and it includes some part of prototype 1. 7

13 c) Testing and Evaluation: Testing and evaluation was done after implementation of both prototypes. In case of first prototype, manual evaluation was done by visualization of possible hypothesis on Volume viewer and comparing it with the output of first prototype. In second prototype a fitness function was used which calculate the RMSE by finding the Euclidean distance between the points of model points and target points (possible hypothesis). d) Write-up: The write-up was done as the project progressed. The first part of write-up was delivered via an interim report Milestones Milestones were added to the project plan so that the progress of project can be measured at any point during the project. Each milestone is denoted by a number in the Figure 1.3 above and the details of those milestones are listed below: 1. Preparing dataset s (week 17) 2. Complete interim report (17 June 2011 week 18) 3. Completing implementation of prototype I (week 21) 4. Evaluating prototype 1 (week 22) 5. Completing implementation of prototype 2 (week 27) 6. Evaluating prototype2 (week 29) 7. Finalize the report (week 29) 8

14 Chapter 2: Background 2.1 Literature Review What is Segmentation? Segmentation is a process of partitioning a digital image into multiple segments. The goal of this process is to simplify and/or change the representation of image into something more meaningful and easier to analyze [2]. There is a very important role of segmentation in my project because main objective of this project is to automate the segmentation of an airway which clinicians do manually. This process assigns a label to every pixel in an image such that pixels with the same label share certain visual characteristics. Outcome of segmentation is a set of segments which cover the whole image Image Segmentation Methods: Several general-purpose techniques have been developed. Some of the techniques relevant to my project are as follows: Data based approaches Thresholding: Thresholding is the simplest method of image segmentation. In this process a threshold value is applied and each individual pixel of image is marked as an object pixel, if their value is greater than threshold value they are known as threshold above and if below than Threshold Below. Threshold above are given value 1 and to other we give value 0 [2]. In the first step of implementation of this project, we select a suitable threshold value to convert the Grey image into a Binary Volume. Airway and lungs are much less dense than the surrounding tissues; therefore a simple threshold can separate these areas nicely Region growing method: Region growing is a simple region-based image segmentation method. This method of segmentation examines neighboring pixels of initial seed points and determines whether the pixel neighbors should be added to the region. Many existing airway/vessel segmentation methods, which I will explain in the next section, make use of the idea of region growing segmentation techniques. There are varieties of image segmentation techniques in the literature. A new region growing method for finding the boundaries of blobs in presented in Region Growing: A New Approach [3]. In this method, at each step, at most 9

15 one pixel exhibits the required properties to join the region. This method uses two novel discontinuity measures, average contrast and peripheral contrast, to control the growing process. Average contrast is the difference between the grey level average of the current region and the average of its current boundary (CB). Peripheral contrast is the difference between the grey level averages of the current boundary and the average of the internal boundary. For the citation of work done in same domain of my project, understandability of this method is very necessary Level Sets: Level set method use a partial differential equation (PDE) based method and by solving these PDE equation image segmentation can be done. This method was initially proposed by Osher and Seithian in 1988 [4]. This method is very effective to address the problem of curve, surface etc. This method has number of advantages for example it is parameter free, provides a direct way to estimate the geometric properties of evolving structure. This method has been used in segmentation of 3D tumor in Magnetic resonance images [5] because of its ability to handle complex geometry and topological changes and its computational efficiency in 3D Model Based Approaches Point Distribution Model In this project we are trying to model the airway. Models of shape are used widely in computer vision using a priori knowledge of object shapes; The Point distribution model is one of such model. An object is modeled in terms if landmark points positioned on object features, and at regular intervals in between. After identifying such points on a set of training examples, PCA can be used to discover the mean object shape and the major modes of shape variation [6]. Principal component Analysis (PCA) is a procedure that converts a set of observations of possible correlated variables into a set of values of uncorrelated variables called principal components. The number of principal components must be less than or equal to number of original variables [7]. Tutorial of Jonathod Shlens [7] on Principal component Analysis helped a lot in understanding the basics of PCA. He has explained the mathematics behind PCA in a very simple way. The Active shape model (ASM) [8] is a statistical model of the shape of object which iteratively deforms to fit to an example of the object in a new image. These shapes are constrained by a Point distribution model so that they can vary only in ways seen in a training set of labeled examples. This can be one of the ways to achieve the solution, by preparing some training images by labeling CT images manually. The ASM algorithm will match the model to a new image by alternating following steps: Look in the image around each point for better position for that point 10

16 Update the model parameters to best match to these new found positions. In another paper [9] the same method s practical implementation is shown by using example of resistor shapes and hand shape Pictorial Structural Models: Pictorial Structural models, introduced by Fishchlet and Elschlager [10], provide a statistical model of objects. This model base approach can be used to recognize and locate the objects in an image and it includes a probabilistic approach to the training and recognition of pictorial structure models in image. This process involves the maximization model score which depends on two factors: the score of each part and the location of the parts. To perform this maximization against the image, the model must be trained and the model space must be searched to find the best configuration of the model given the image. Pictorial Structures for object Recognition [11] is motivated by the pictorial modeling and recognition of objects. They have created models to represent faces and human bodies and using those models to locate the corresponding objects in novel images Data Pre-Processing Methods: Connected Component Analysis Connected component analysis scans a binary image and group its pixel based on pixel connectivity. Connected-component labeling is used to detect the connected regions in binary images. It works by scanning an image, pixel by pixel, and identify connected pixel regions. Connected component analysis can be used after the Thresholding of data and If we make an assumption that lungs and airways are connected, simple statistics can be used to find out the biggest connected component, which will be airways and lungs Thinning Thinning is a process which removes layers of pixels on boundary, in such a way that we get a skeleton of singular pixel width [12]. Following are the 3 important methods of thinning: a) Sequential thinning: In this method, I iterate through the set of pixels, flag certain pixels as boundary pixels and at the end of iteration I delete the pixels which are flagged. 11

17 b) Parallel thinning: This method examines pixels for deletion based only on the results of the previous iteration. c) Non-iterative thinning: Minimal distance of a pixel to the edge of the containing object is used to extract the skeleton in noniterative thinning[13]. According to the measurements[13] this method is faster than other known sequential algorithms. To create a tree model of input data, we have to apply thinning on the binary data we have. This will provide us a graph and after connecting pixels of graph, we can use various methods to get a tree. Approach used to get tree in this project is Minimum Spanning tree Minimum Spanning tree A spanning tree of a graph is a sub graph that contains all the vertices and it is a tree. One graph can have many spanning trees. If we assign weight to each edge and assign a weight to a spanning tree by computing the sum of the weights of edges than Minimum spanning tree is a spanning tree with the minimum weight. There are two algorithms commonly used, Prim s algorithm [14] and Kruskal s algorithm [15], both of them are greedy algorithm and run in polynomial time Registration of point sets Registration is the process of transforming sets of data into one coordinate system from another to align the content. In this project CT images of lungs can be of a child, women or men and depending on the height and other attributes, the size of airways can be different from each other, so alignment of the training model was necessary. Before applying the statistical analysis on the shape models it is important that the shapes represented are in the same co-ordinate frame. One of the approaches to align the shape model is presented in A Method for registration of 3-D shapes [16]. This method is based on the iterative closed point (ICP) algorithm, which requires only a procedure to find the closest point on a geometric entity to a given point i.e. doesn t require correspondence. Another method is Procrustes Method [17] in the statistical analysis of shape, which is performed by optimally translating, rotating and uniformly scaling the objects and aim is to obtain a similar placement and size, by minimizing a measure of shape difference called the Procrustes distance between set of objects. This methid require 12

18 correspondence. Another method was presented by Horn[18] using unit quaternion to represent the rotation. This method required 1:1 correspondence. 2.2 Previous related work Vessel Segmentation: Airways and vessels are similar in structure so in area of research they are linked closely. They have a lot of same characteristics i.e. tubular in shape, tree-like structure, with an outer wall encasing the lumen. Research done in this area can be divided into: a) Segmenting the vessels b) Modeling the vessels I have cited two papers in this domain. Blood vessel segmentation using moving-window robust automatic threshold selection [19] posted the idea of a moving window based Thresholding method of the segmentation of blood vessels in human brain. Another citation Threshold selection based on a simple image statistic [20] is about a novel method for automatic threshold selection, based on statistics, which can be acquired without making histogram of grey levels of the image. Dynamic Thresholding is a very novel and interested idea. I feel that both methods would be unnecessary in my application as It will be better to take the image volume as a whole 3D than the sequentially scan through smaller parts. Also there is no need for dynamic Thresholding because we can use simple threshold to remove background. Instead of using dynamic programming I have used Volume viewer to threshold the data and after some variation in threshold values I have selected a single value of threshold which can successfully threshold all datasets Airway Segmentation: Plenty of work is done in the field of Airway segmentation for various possible purposes such as medical research, diagnosis of lung-related illnesses and monitoring effects of treatments. A differential geometric approach to automated segmentation of the human airway tree [21] outlines a method for generating a 3D model of the bronchial trees based on CT images. This algorithm firstly acquires a basic segmentation of the lungs thorough Thresholding and then applies marching cubes to model the anatomical structure. Marching cubes produces a triangle mesh by computing iso-surfaces from discrete data. It processes the mesh to remove the non-tubilar bits and fill holes. By connected 13

19 patches of all cubes on the iso-surface boundary, it gives a surface representation. This system is capable of generating some impressive and indeed realistic looking 3D models of inner working of lungs but the accuracy of the method comes into question because inability to analyze it against a ground truth. This method creates a visualization of the airways as opposed to providing an actual model to be used for pruning. Segmentation and quantitation of the primary human airway tree [22] uses a 8- connected region growing technique to follow the downward branches of the tree, the detection of upward branches works in the same way. The key Problem with this method is leakage. Segmentation and Airway Morphology analysis from Low-Dose CT scans [23] is a more recent research article. The method uses segmentation through fuzzy connectivity focused on a relatively small region of interest which follows the airway as it is being segmented. The actual detection of leaks is achieved through the knowledge that a leaked segmentation result exhibits a spongy structure i.e. it is full of holes. This method is very impressive in theory but accuracy achievable by this method is questionable. Segmentation and analysis of the human airway tree from three-dimensional X-ray CT images [24] is the most relevant work to my project because it is using a model base technique to segment the airways. In this work, 3D airway tree is constructed by segmenting the airway tree using grayscale morphological reconstruction. This is a fully automatic technique for segmentation of airway tree in 3D CT images. The primary difference in this work and my project is that, they have 3D CT images which are of very high quality, so by using simple segmentation they can get the airway out of CT image. In my case we have a 4D CT dataset so the quality of images is not as high of 3D and if we use the same approach, it will result in leaking problem. In this method they are not using a model approach which I am going to adopt. Instead of using data based segmentation approach, as done in this method, I will fit a model to find out the airway tree out of CT image. Model base segmentation can be the best way to avoid leakage problem and even if we have leakage we can simply tell that the model is unable to fit. 14

20 Chapter 3: Design 3.1 Overview To understand the design of this project, we can divide the design in 2 parts. First part includes the analysis of data, creation of training tree model by manually annotating the splits of airways and converting the input data (CT scan images) into a tree using various methods from the computer vision literature. This part of project prepares the ground for applying various approaches to extract the airway. First part, as explained before; provide us a minimal spanning tree as an output which has airways and lungs. In the next part two different approaches were used to extract the airway from the tree. One approach was using dynamic programming and the other one was using the Point distribution model. To test both of the approaches a training tree model was required which was created by manual annotation. The overall design of project is discussed in detail in following sections. This chapter will only explain the design of the project and few things will be difficult to understand because of the complexity. Next chapter will provide the implementation of this design and it will help to improve the understanding of this design with the various output images. 3.2 Data Analysis: There were 6 different datasets available and the first task was to understand the structure of data available and how the splits of airways work. This was done by visualizing all the data sets using various tools. Volume Viewer was the most helpful tool in this whole project because it is very user friendly and light weight tool to load the datasets and visualize any phase of a Dataset. As a first step, all datasets were visualized to identify the similarities and differences between them, as all datasets belongs to different patients. CT scan of a patient is done at different phases of the breathing cycle. After a CT scan, the dataset which is available has various numbers of 3D frames in each temporal phase. One phase usually has 144 frames but it can vary depending on the CT scan. A total 12 phases are recorded on the basis of exhaling or inhaling percentage of lungs. In Figure 3.1 we can see how one of the dataset has 145 frames per phase and each phase s Series Description tell us about the condition of lung. 15

21 Figure 3.1 Screen shot from Sante DICOM Viewer showing the Series Description for each phase. [In =Inhale and Ex=Exhale] As all datasets belongs to different patients and patients can have different height, weight or other different physical attributes so datasets are usually different from each other, as can see in Table 3.1. Currently clinicians/doctors only mark one phase out of 12 phases of a patient s CT scan. After consultation with sources in St James Hospital we came to know that clinicians/doctors mark the phase at 40% exhale. Dataset # Slices per Comments phase Small Lungs and normal number of frames per phase Very big lungs and some extra frames recorded Small lungs and normal frames per phase Normal Lungs and normal frames per phase Normal Lungs and normal frames per phase Normal Lungs and normal frames per phase Table 3.1 Comparison and analysis of available Datasets CERR, which is free tool, can be used to visualize the manually segmented/annotated datasets, as shown in Figure

22 Figure 3.2 Visualization of marked phase using CERR (area inside the green boundary is an airway) 3.3 Data Preparation: After the analysis of data, the next phase was preparation of data for various operations. As one dataset has 12 phases, Volume Viewer was used to generate hdr and img file of each phase. The purpose of generating these files was to load them using ITK toolkit and apply various ITK filters. Instead of using all phases of dataset I have been using only 2 phases from each dataset i.e. 0% EX and 80%EX. I selected these 2 phases randomly so that I have to prepare less data and spend more time on the actual aim of this project. In total there was 12 phases for each patient whom I used to load for segmentation and I used same phases for taking manual annotations. 3.4 Generation of a Tree model from image data: Thresholding Connected Component Analysis Thinning Graph Tree After analysis and preparation of required input data the next step was the segmentation of input image data and getting an output in form of a tree so another training tree model can be fitted on top of this tree to identify the airway. This work was already done by Derek Magee and it was very important to understand all this process. First step of this process is to Threshold the input data and create a binary 17

23 volume. A fixed threshold value was used for his purpose and this threshold value was selected by varying different threshold values and analyzing the effects using Volume Viewer. After Thresholding the dataset, connected component analysis was applied on the binary data to get only airways and Lungs from the data and remove the background. Connected component analysis provides all the possible connected components, using statistics the biggest connected component was selected; which was airways and lungs. This was done by making an assumption that airways are always connected to the lungs. This helped to remove the background and get only the airways and lungs. Now Skeltonization [12] was applied on the outcome to get the thinned airways and lungs. By connecting the thinned pixels the available data was converted into a Graph and then a tree was obtained by converting the graph to tree using Minimal Spanning tree. This all work was done by the ITK filters and it provides very good results. Tree obtained as an output only has airway and lungs and in next part I will explain how different approaches were tried to extract only the airways from the whole tree and remove the lungs. Every new dataset is converted into a tree using the above approach to identify the airways. 3.5 What is a model? After getting a minimal spanning tree, we need a model which can be used to extract the airway from that tree using dynamic programming or Point distribution model. A model is actually the structure of airways which is obtained from manual annotation of dataset, after loading it into Volume Viewer. In figure 3.3 you can see a simple 4 point model. Figure 3.3 A simple 4 point model 18

24 3.6 Generation of Hypothesis The Minimal spanning tree provides a tree model of the input data. The currently implemented system provides the tree as an output in a comma-separated values (csv) file. The number of nodes in an output tree can be reduced on the basis of the distance between each node and that node's root; this was used to generate the hypotheses. Different distance values were used to generate the tree depending on the size of the model. For a 4-point model a small distance value was used to produce an small output tree and for a 9-point model a comparatively bigger value was used. To generate the hypotheses, the software Volume Viewer was used (see Sect. 1.6). Volume viewer can load annotations and show them on top of the loaded data. The hypotheses were then generated manually (see Chapter 4). Nevertheless, it is possible to automate the hypotheses-generation step; methods to this were explored, but given the limited research time and the complexity of the process itself, this topic was not fully researched and hence not implemented. 3.7 Manual annotation of points: A second training tree model was needed after generating the initial tree from the input data. This tree model can be fitted on the tree to extract the airway. As with the hypotheses, the training tree models were generated manually using Volume Viewer. Some other tools are also available to draw annotations (e.g. Slicer, a free, open source software package for visualization and image analysis) but Volume Viewer was used for simplicity. When annotating on the images two key aspects need to be understood: (1) the splits in the airways, and (2) the timing of such splits. Each marked phase of the dataset was loaded into CERR to understand the airways' splits for that particular dataset. In a preliminary phase only 4-point training models were generated. Each the 6 different available datasets consists of 12 phases and hence a total of 72 (6 times 12) phases were available. Only two phases were used, mainly because: 1- There is very low variability in the splits within a same dataset. Thus, considering all the phases would not provide much more additional information but rather would have unnecessarily increased the number of training models. 2- By considering fewer phases, the process of manually annotating the datasets was done more rapidly, thus optimizing the research time. 19

25 At later stages of the project, the preliminary 4-point training models were extended to a 10-point model for final testing the implemented system and hence assessing the overall outcome of the project. 3.8 Aligning the Training model As discussed before, the available datasets are different one from each other and hence the tree models derived from each different presented substantial difference; therefore, all the models needed to be aligned to the same plane. This was done using generalized Procrustes analysis [17]. Randomly, one of the shapes was considered as a reference shape and all other shapes were aligned according to that reference. A visual demonstration of the shape-alignment process is presented in Chapter Prototype 1: Dynamic programming The first prototype of this project consisted in evaluating and enhancing an existing method implemented by Derek Magee. The method uses dynamic programming to fit the model's points to a previouslygenerated minimal spanning tree, to finally extract the airways. Training models were created by manually annotating the datasets and various other parameters were calculated i.e. Standard deviation of x, y and z values. Direction of each point in the model (unit vector). Length of each edge and Standard deviation of the Lengths. This prototype calculates the probabilities of the leaf nodes and then searches for the best child (i.e. the one with the highest cumulative probability). To calculate the cumulated configuration probability, the algorithm loops through all the possible children to evaluate the model's edge probability, and then calculates the cumulated probability along the edge and the direction probability. Some heuristics were applied when fitting the model i.e. 1- Child and parent can t be the same node. 2- The child can t be closer to the root than the parent. 3- Exclude the nodes where the defined distance from the root is less than maximum node depth. 20

26 3.10 Prototype 2: Point Distribution Model Figure 3.4 (a) dependencies of nodes in dynamic programming Figure 3.4 (b) dependencies of nodes in PDM If we have longer sequences, dynamic programming is good approach to use. But in my case dynamic programming was not producing the desired results. In Point distribution method, nodes of tree model can have better understanding as compare to the dynamic programming. As we can see in Figure 3.4 (a) that leaf nodes of tree model have link with their parent node but they don t have any link between them. Iteration 1: The same manual annotations used in the first prototype of the project (Sect. 3.9) were used as landmarks in this prototype. For simplicity, only 4-point training models were considered in the first iteration. Since the datasets are three-dimensional, the resulting shape model will be 12-Dimensioned (3 dimensions times 4 points). Training annotations were first aligned using generalized Procrustes analysis (Sect. 3.8), which minimized the least squares error between the points. The Principal Component Analysis (PCA) was then applied to estimate sensitivities of the shape model to the different dimensions and points. The number of principal components (PCs) was reduced by analyzing the Eigen values returned by the PCA: only those PCs with Eigen values greater than zero were considered, as varying those Eigen values would produce very little or no effect on the shape model. As a result of the PCA, the mean shape of the model and a number of Eigen vectors (and corresponding Eigen values) were obtained. The main modes of variation of the training shape (i.e. locations of the objects within the training set) were then analyzed to measure the effect of the different Eigen vectors as described by. If X is an instance (i.e. a variation of) of the shape model so we can represent X by Eq

27 X={ x 1,y 1,z 1,x 2,y 2, z 3.. x n,y n,z n } (1) where (x i,y i,z i ) is the position of the i th point. New shapes can be approximated using Equation 2. X = Ẍ + Pb (2) Where Ẍ is the mean shape derived from the PCA, P (Equation 3) is a matrix formed from the first t selected eigen vectors, and b (Equation 4) is a vector of weights. P = (p 1,p 2,p 3 p t ) (3) b = (b 1,b 2,b 3. b t ) T (4) To generate the new shapes these vectors of weights can be varied within an appropriate limit in Equation 2: as the variance of the i th parameter, b i, across the training set is given by λ i, by limiting the parameter b i to ±3sqrt(λ i ), all the new shapes generated would be statistically similar to the original training shapes. A clear demonstration of this shape variation is done in Chapter 4. Once the PCs were selected and the limits within which the new shapes can be generated, the shape model had to be varied (within the defined ranges of b) to test the various hypotheses (currently manually generated from the output of the Minimal spanning tree) on it. A simple iterative approach as described by Cootes [25] for matching model points to target points is used as follows: 1- Initialize the selected shape parameters to 0 which Is b i =0,b 2 =0,b 3 = Generate a new model using values of P and b X= Ẍ +Pb 3- Transform the target points to the new model points using Procrustes analysis. 4- Update the model parameters to match to aligned new points. b=p(ῡ - Ẍ) where Ῡ are the aligned new points 5- If not converged, return to step 2 This approach usually converges in a few iterations (usually within the first five). After finding the closest model shape for the new aligned points, a goodness-of-fit measure was required to test whether the new shape is similar to the model shape. The Root Mean Square Error (RMSE) was calculated between the model (i.e. the mean shape) and the hypothesis by calculating the Euclidean distance between each model's nodes to each hypothesis separately and finally calculating the total deviation. 22

28 Iteration 2: After developing the system with a 4-points model, a 10-points model was used. By increasing the number of nodes, the dimension of shape model was increased to 30 (10 points times 3 dimensions). The same approach as in iteration 1 was used, but this time with more points in the shape model. After applying the PCA and analyzing the PCs and eigen values, eigen vectors with no-significant influence on the shape model were discarded (i.e. those with eigen values close to zero). New hypothesis were generated from the minimal spanning tree, this time with equal number of points as the new shape model (i.e. 10 points). Iteration 3: The first PC in the 4-point model of the first iteration and the 10-point model of the second iteration explained a large proportion of the model's variance, thus the shape model was found highly sensitive to the changes in the first eigen vector's weight, but rather insensitive to the variations in the other eigen vectors' weights. The first point in the model represents the starting point of the airway and the second point is the first split of the airway tree. After the deep analysis of the datasets it was found that the reason of these sensitivities was due to one of the datasets, which presented extra frames in every phase. This dataset affected the position of the first point in the shape model, as all the other datasets showed their first points from the first frame of each phase. Instead of discarding that dataset or making changes on it, changes were done in the shape models by removing the first point from all of the training shape models. This did not affect the purpose of this research, as in the airway tree the first point is very obvious and the most relevant points are the points where the airway starts splitting. After removing the first point, if the splits of the airways can be extracted from the output of minimal spanning tree, the first point can be sought using simple heuristics. After removing the first point from all the shape models and generating new hypothesis manually, this time with 9 points, the same approach as described above was applied. Results can be seen in Chapter 4. 23

29 Chapter 4: Implementation 4.1 Understanding Datasets: The most important and basic part of this project was to understand the structure of the airways in the different datasets and how the clinicians/doctors mark the airways in the CT images. As explained before, one of the phases of each dataset was already marked and can be loaded using CERR for visualization. After saving the marked phase from the dataset using Sante DICOM Viewer, that phase has a file with the information of the marked airway. This information can be loaded in Matlab using the DICOM parser. Figure 4.1 Actual marked phase plotted as a tree using Matlab (left) versus Drawing by Dr Kevin N Franks (St James's Institute of Oncology) (right) The drawing by a clinician from St James s Institute is highly related to the plotting of a given marked phase using Matlab, and considerably helped in the understanding of the airways structure (Figure 4.1). The drawing helped a lot on understanding the splits of the airways, and their importance to clinicians. 24

30 4.2 Aligning Training Models: As mentioned before, the manual annotations from each dataset are different one from each other and hence and aligning of all the models according to a mean reference shape was needed. This was done by using the generalized Procrustes Method in Matlab (see Sect. 3.8). Figure 4.2 (a) Before Alignment Figure 4.2 (b) After Alignment Figure 4.2 Result of the process of alignment. (a) Non-aligned shapes, and (b) aligned shapes. A 4-point model was considered as a mean reference shape and all other points were aligned to that particular shape. Very low variability was found in two of the training models (a). This is because those 2 training models belong to the same dataset, hence justifying the decision of annotating only 2 phases of each dataset instead of all of them or more than 2. Using 10-point models, alignment via Procrustes analysis is also possible and was observed to yield very accurate results, even for cases where a relatively high variation among shapes was present (Figure 4.1). 25

31 Figure 4.3 (a) Before Alignment Figure 4.3 (b) After Alignment Figure 4.3 Result of the process of alignment. (a) Non-aligned shapes, and (b) aligned shapes. A 10-point model was considered as a mean reference shape and all other points were aligned to that particular shape. 4.3 Generation of the Minimal spanning Tree Figure 4.4 Visualization of 4 different frames of input data ITK filters [26] were used for the segmentation of input data. In the first step an.img file, as shown in Figure 4.4, as generated from volume viewer was read and the ImageFileReader filter was used for this purpose. "ImageFileReader" is a general filter to read data files. In the next step, the input data was threshold using a constant threshold value. BinaryThresholdImageFilter" is an ITK s filter which 26

32 binarises an input image by Thresholding. This filter produces an output image whose pixels are either one of two values (Outside Value or Inside Value) and requires lower threshold value and upper threshold value. The lower threshold value was set to (the minimum possible data value) and upper threshold value was decided by varying the threshold value and observing the outcome using Volume Viewer. After Thresholding the input image file, connected component analysis was applied to derive the airways and lungs from the input image data. ITK's filter BinaryImageToShapeLabelMapFilter, which converts the binary image to a label map and valuate the shape attributes was used to that aim. We have to set a foreground value and background value for this filter with some other attributes. The foreground value was set to 255 whereas background was set to 0, assuming the background will always be black. This filter generates a labeled map that was later converted into a labeled image using a filter named LabelMapToLabelImageFilter. Using some statistics the biggest component was finally selected. This produced images with only airways and lungs (Figure 4.5). Figure 4.5 Output of connected component analysis, showing various frames, with only airways and lungs. Next, Skeltonization was applied to produce a skeleton of each pixel width and to remove the layers of the boundary pixels. Thinning filters of ITK were used for this purpose (Figure 4.6). 27

33 Figure 4.6 various frames of a phase after applying the thinning filters compare to Figure 4.5 By connecting the points of thinned the volume, a graph was created and then converted to a minimal spanning tree using Kruskal algorithm [27]. The minimal spanning tree was then saved in a CSV file and then loaded into Matlab for visualization (Figure 4.7). Figure 4.7 phase 0% EX of dataset # 6 as a tree In some of the datasets, only one airway and one lung rather than both lungs are produced, mainly due to the noise in the data. Also, while doing the segmentation, lungs and airways are not connected; however, 28

34 this was not a problem for this project as the airways are the main concern. Tree visualization of first phase of each dataset can be seen in Appendix C. Volume Viewer provides a facility for loading the annotations and annotates the dataset with those points. The minimal spanning tree was displayed on the same phase of the dataset to see the results (Figure 4.8). This was later used in the manual generation of hypothesis. Volume Viewer provides a facility for loading the annotations and annotates the dataset with those points. Minimal spanning tree was displayed on the same phase of dataset to see the results. This visualization of tree, as shown in Figure 4.8, helped in the generation of manual hypothesis. Figure 4.8 annotating output tree on the dataset. 4.4 Prototype 1: Dynamic Programming In a previous version of this prototype, handpicked values for model training were used. As a first step, all the manual annotations were loaded into the same prototype and calculated all required parameters using C++ into the same code. After creating a tree model from the training models, that model was fitted on the various datasets and the output was recorded. This prototype provides the new model points fitting on the minimal spanning tree. Very little implementation work was done on this prototype of project and most the work was related to the evaluation of output, presented in Chapter 5. 29

35 4.5 Prototype 2: Point Distribution Method Generating Principal Components: First step of this prototype was to find out the PCs of the training shape models. This part of project was implemented using C++ partly using a code provided by Derek Magee. This code uses clapack to calculate the Eigen vectors and Eigen values. Iteration 1: The 4-Point Model In the first iteration, a 4-point simple model was considered so that system remains easy to analyze. Analyzing the Eigen Vectors: Few important Eigen vectors on the basis of descending-order Eigen values were selected (Figure 4.9). 120 Eigen Values (4-Points) Eigen Values Figure 4.9 Eigen values of 4-point model As a 12-Dimensioned shape model was produced in the case of a 4-point model, 12 Eigen values were available by the PCA. Figure 4.9 clearly shows how only first 4 Eigen values are important and rest of them are almost zero. In this case, only the first 4 Eigen vectors were selected and the rest of them were ignored. Variation in shape models: After selecting the relevant Eigen vectors, it was important to visualize the available variation in the shape models to identify the sensitivities of the mean training shape to the different Eigen vectors. This 30

36 variation was done in ±3 s.d of the first 4 model shape parameters so that the new shapes are similar to the training shapes (Figure 4.10). Figure 4.10 (a) Mode 1 Figure 4.10 (b) Mode 2 Figure 4.10 (d) Mode 3 Figure 4.10 (c) Mode 4 As the Eigen values were in the descending order, the variation in the shape of Figure 4.10 (a) is far greater for the first Eigen vector (Figure 4.10a) than for the other 3 variations. 31

37 Fitting model on hypothesis models: As explained in Chapter 3 (Design), after finding the principal components, the next major task was to match the model points to target points (Manually-generated hypothesis). The method used usually converges in the first 3 iterations (Figure 4.11) Correct Hypothesis: Figure 4.11 (a): Iteration 1. Figure 4.11 (b): Iteration 2 Figure 4.11 (c): Iteration 3 The outcome with a correct hypothesis would be a quickly-converging system (Figure 4.11). In Figure 4.11 (a) the blue model is the mean shape, the red model is the new hypothesis and black model is the transformed form of the hypothesis. 32

38 False Hypothesis: Figure 4.12 (a): Iteration 1. Figure 4.12 (b): Iteration 2 Figure 4.12 (b): Iteration 3 In Figure 4.12 (a) (b) (c) we can see that if the hypothesis is not correct, after little iteration the shape model will be very different from the hypothesis model. This happen because the shape model can only be changed within a limit and that limit make sure that the shape model remains same as of training shape models. 33

39 Iteration 2: The 10-Point Model As beforehand stated, a bigger model was used in iteration 2. Using the same procedure as used in the first iteration, PCs of the shape models were calculated (Figure 4.13). Analyzing the Eigen Vectors: 350 Eigen Values (10-Points) Eigen Values Figure 4.13 Eigen Values for a 10-points model In iteration 2, the first Eigen value explained up to 90% of the overall variance in the 30-dimensional dataset (see Chapter 3). By changing the shape model using Mode 1 (i.e. first Eigen value), it can clearly be seen that it is mostly affecting the first point of shape model (Figure 4.13). 34

40 Figure 4.14 High variations within the first point of model Iteration 3: The 9-Point Model In iteration 3, the first point of each dataset was removed from the shape models and PCs were calculated again and the same procedure as for iteration 1 was followed. In this iteration also, only the first 4 principal components were considered because they showed significantly high values (Figure 4.15) 60 Eigen Values (9-Point Model) Eigen Values Figure 4.15 Eigen values for 9-Point model 35

41 After selecting the relevant principal components, the same technique was used to match the target points (Figure 4.16,a, b, c). When comparing this shape model with a correct hypothesis, the shapes were observed to converge in very little iteration (Figure 4.16). Figure 4.16 (a) Iteration 1 Figure 4.16 (b) Iteration 2 Figure 4.16 (c) Iteration 3 In Chapter 5, this method and correspondent implementation was evaluated for applicability. 36

42 Chapter 5: Evaluation 5.1 Prototype 1: Dynamic Programming In prototype 1, from manual annotation, a training model was created and that model was fitted on the output of the Minimal spanning tree in order to derive the airway from the tree. Output of this prototype was in a form of a model which could be the correct answer or a miss-fitted tree. A manual adoption method was adopted to evaluate the output of this model. The minimal spanning tree provides the all points of a given tree, including all ends and junctions. Matlab was used to plot the ends and junctions of both the tree and the fitted model. After plotting all the points, ends and junctions were also loaded on Volume Viewer to test the hypotheses. The leave-one-out evaluation method was used for testing the outputs. In this method, one of the datasets is not used in creating the training model and the minimal spanning tree of opts out dataset is then created to extract the airways. The result of this method was inaccurate as it only succeeded in 2 out of the 6 datasets (33.3%). Only a 4-point model was used to keep the system simple for testing. As the desired results were not achieved, even with a simple 4-point model, no more evaluation was done for this prototype and the rest of the time was spent on prototype 2. 37

43 Figure 5.1 Output of the one leave out validation. Model with green color is the correct hypothesis and model with red color is the fitted points I got as an output of prototype 1. Correct answer can be seen in Figure 5.1 (b) and Figure 5.1 (c). Figures (a), (d), (e) and (f) correspond to wrongly-fitted models. 5.2 Prototype 2: Point Distribution Method: To evaluate the output of prototype 2, fitness functions needs to be defined. Once the shape of the shape model is varied and the shapes stop converging, it means the shape is as close as possible to the hypothesis. A numerical assessment of this "closeness" is required. Using volume viewer a correct tree model (i.e. a hypothesis) for an airway can be created. This correct hypothesis and the tree model from a shape model would have same number of nodes. The fitness function used calculates the Euclidean Distance between every point to finally compute the RMSE. 38

44 5.2.1 The 4-Point model: As a 4-point model was initially used, the evaluation of this method was not extensive. Using a simple validation and once the desired results were obtained, the size of the model was increased up to 10 points (and later reduced to 9 points) in order to achieve the objectives of this project. The same evaluation criterion was used for both models. Three different hypotheses were manually generated by loading the minimal spanning tree overlaid on the respective phase of dataset in Volume Viewer. For example, one of the phases of dataset#5 was not used in the calculation of principal component, but all the phases from all the other phases were used to calculate the PCs. Next, by loading the left out dataset to Volume Viewer and loading the minimal spanning tree as an annotation, 3 different hypotheses were selected. The first hypothesis was correct; the second one was nearly correct and third one was a totally wrong hypothesis. An iterative fitting as described in Chapter 4 was used to fit the model to the hypothesis TRUE Nearly true FALSE Iteration 1 Iteration 2 Iteration 3 Iteration 4 Figure 5.2 Testing hypothesis with 4-point model Using three different hypotheses with 4 iterations, it can clearly be seen (Figure 5.2) that the substantial reduction in the RMSE from the first to the second and next iterations; this is because the hypothesis are not aligned with the mean shape. After the 2 nd iteration, there is a very small change in the RMSE 39

45 meaning that a greater number of iterations is not required. The method was found to work for this configuration; thus, if a considerable number of hypotheses is generated from the minimal spanning tree, this method can be used and the hypothesis with the least RMSE can be selected as the correct hypothesis. So in this way the position of the airway in the minimal spanning tree can be found. Now in the next step, the same approach for bigger model will be used The 9-Point model: As only 6 datasets were available for this project and in all 12 phases within a same dataset are very similar to each other, in 9-Point model only one phase of each dataset was considered. A Leave-one-out cross validation was then done. The evaluation results of this method were very good but time consuming and expensive [28]. No evaluation of the 10-point model was done because of the assumption that a simple heuristic can be used to re-locate the first point and hence it could be removed from the training sets Iteration 1 Iteration 2 Iteration TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE Opt out 1 Opt out 2 Opt out 3 Opt out 4 Opt out 5 Opt out 6 Figure 5.3 Testing hypothesis using 9-point model The process of evaluation was very time consuming as for each dataset a correct and wrong hypothesis was manually generated for each case (Figure 5.3). Correct and in-correct hypothesis for dataset #1 can be 40

46 seen in Figure 5.4(a) and Figure 5.4(b). From the Figure 5.3 it can be seen how after each iteration, the RMSE of the True (i.e. correct) hypothesis decreases and on the other hand the false (wrong) hypothesis has a very big value of RMSE that keeps nearly constant as more iterations are done. Figure 5.4 (a) correct hypothesis for Dataset # 1 Figure 5.4 (b) in-correct hypothesis for Dataset # 1 In the next step Leave two out validation was done. In this process, 2 phases of each dataset were used and the complete dataset was opt out and then the correct and false hypothesis for both phases of the opt out dataset were generated. Same approach, as used in the Leave one out validation, was used and the averages for RMSE for each dataset were computed (Figure 5.5). Results of Leave two out validation are almost similar to leave one out validation and it indicates that if all of the 12 phases of the training data are used for calculating PCs, the airway from all phases of a new dataset can be extracted. Due to time constraints, it was not possible to annotate all of the phases manually, hence only 2 phases were used. 41

47 Iteration 1 Iteration 2 Iteration 3 0 TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE Opt out 1 Opt out 2 Opt out 3 Opt out 4 Opt out 5 Opt out 6 Figure 5.5 Average RMSE for each opt out dataset 42

48 Chapter 6: Conclusion and Future work 6.1 Conclusion The basic aim of this project was to extract the airway from the lungs so that we can fully or partially automate the process of marking the airways on the CT scan images and save a lot of time of clinician. First prototype used was not a success, as we have seen in the last chapter. This prototype works on datasets out of 4 with the simpler model so I decided to move to the next approach. Point Distribution model worked very well with the simple model and as well as with the bigger model. Results of leave one out validation and leave two out validation shows that this system works well with the unseen data. Further testing of this system is required and it will only be possible if we have a search technique implemented, which can generate all possible useful hypothesis as it is difficult to specify the hypothesis manually. Once we have a list of possible hypothesis available, the system can choose the best possible hypothesis. Evaluation of 9-point model with leave two out validation shows that if we use 2 phases of each dataset, create training models from 5 datasets and use 2 phases of opt out dataset for validation; we are still able to identify the correct hypothesis. From this evaluation we can conclude that the same system can be used to extract airways from more than 2 phases.. I have tried my best to provide a user friendly Graphical User Interface (GUI) for each and every step of my project to create better understanding and presenting the output in an understandable way. 6.2 Future Work Improved annotations Training models play the key role in this project and it is very important to have correct models. I have done the annotation of datasets by myself and maybe due to lack of experience and knowledge, those annotations have slight error. If a skilled person, who has better knowledge about the airways, can do these annotations to create the training models, it will obviously increase the accuracy of system Increasing the training data Only 6 datasets are used for this project. Increasing the size of training datasets will obviously provide a more representative model. If more training datasets are used, more training model can be created and better statistical model can be created for the extraction of airway. Current available datasets were very 43

49 different from each other and I am sure if we have more datasets, more variation can be found, which will help us to find out more representative principal components. In the evaluation chapter we have seen the results of leave one out validation on the available data and we can see that if we leave out one of the dataset and use others for training, we are still getting very good results and we can find the correct hypothesis for the excluded dataset. So we can say that even with only 6 dataset we have achieved acceptable results but still it is necessary to have more datasets available to improve the system Search Strategy to find possible hypothesis After generation of the Minimal Spanning Tree, currently I was getting the hypothesis manually. This was very time consuming and difficult for me to extract the hypothesis manually. A search strategy is required here, so that we can find out hypothesis automatically. Minimal spanning tree provide us two different types of output, one is the tree with all points of tree and another output is just the ends and junctions of the tree. Minimal spanning trees of various CT scan data can be seen in Appendix B. While generating a minimal spanning tree, we can reduce the size of tree by using a threshold value. For example if I am using a 10 point model, I can limit the depth of my tree to 20. One of way doing this is defining the distance from the root node and then considering all those ends and junctions which are within that distance and ignore rest of tree. This will decrease the size of tree thus the total number of hypothesis. I have spent some time on working to automate this task but to time constraint I couldn t complete it. Random Search [29] or Branch Bound Algorithm [30] can be a possible way of doing this search. As the search space is very big, we need to include some heuristic which can help in reducing the possible hypothesis. For example we can specify a possible distance between the nodes or we can specify the directions Airway contouring and evaluation After finding the correct hypothesis from a minimal spanning tree, next step is how we can go back to segmentation and mark the airways on the DICOM dataset. Threshold of distance and Region Growing [3] can be one of the possible ways to do it. Christopher Parker [31] has tried to achieve the same task using a different approach. He has tried contouring the airways by boundary tracing in each 2D slice [32] and used Hausdorff distance [33] to evaluate the difference. Dice coefficient[34] can also be a possible way of doing the evaluation to find the difference. In current system, similar approach can be used and the selection of 2D components can be done using model. 44

50 References: 1. Levitzky MG (2007) Pulmonary Physiology: McGraw-Hill (Lange Series). 2. Stockman LGSaGC (2001) Computer Vision. New Jersey: Prentice-. Hall. 3. Hojjatoleslami SA, Kittler J (1998) Region growing: a new approach. Image Processing, IEEE Transactions on 7: Sethian SOaJA (1988) Fronts Propagating with Curvature-Dependent Speed: Algorithms Based on Hamilton-Jacobi Formulations. Journal of Computational Physics. 5. K. Aloui amsn (2009) 3D Brain Tumor Segmentation Using Level-Sets Method and Meshes Simplification from Volumetric MR Images World Academy of Science, Engineering and Technology Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models - their training and application. Comput Vis Image Underst 61: Shlens J (2009) A Tutorial on Principal Component Analysis 8. T.F.Cootes CJT (1992) Active Shape Models - `Smart Snakes'. British Machine Vision Conference Springer-Verlag: pp Graham CTaDHCaJ (1992) Training models of shape from sets of examples: Proc. BMVC92, Springer- Verlag. 10. M.A. Fischler RAE (1973) The Representation and Matching of Pictorial Structures. IEEE Transactions on Computers 22: Felzenszwalb PF, Huttenlocher DP (2005) Pictorial Structures for Object Recognition. Int J Comput Vision 61: Lam L, Lee SW, Suen CY (1992) Thinning methodologies-a comprehensive survey. Pattern Analysis and Machine Intelligence, IEEE Transactions on 14: Neusius C, Olszewski J (1994) A noniterative thinning algorithm. ACM Trans Math Softw 20: Prim RC (1957) Shortest connection networks and some generalizations. Bell System Technical Journal. 15. Kruskal JB (1956) On the Shortest Spanning Subtree of a Graph and the Traveling Salesman Problem. Proceedings of the American Mathematical Society Besl PJ, McKay HD (1992) A method for registration of 3-D shapes. Pattern Analysis and Machine Intelligence, IEEE Transactions on 14: Goodall C (1991) Procrustes Methods in the Statistical Analysis of Shape. Journal of the Royal Statistical Society Series B (Methodological) 53: Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. Optical Society of America A Wilkinson MHF, Wijbenga T, de Vries G, Westenberg MA. Blood vessel segmentation using movingwindow robust automatic threshold selection; Sept pp. II vol Kittler J, Illingworth J, Föglein J (1985) Threshold selection based on a simple image statistic. Graphical Models /graphical Models and Image Processing /computer Vision, Graphics, and Image Processing 30: Jiantao P, Fuhrman C, Good WF, Sciurba FC, Gur D (2011) A Differential Geometric Approach to Automated Segmentation of Human Airway Tree. Medical Imaging, IEEE Transactions on 30: Chiplunkar R, Reinhardt, J. M. & Hoffman, E. A. (1997) Segmentation and quantitation of the primary human airway tree. Proceedings of SPIE Medical Imaging,

51 23. Tschirren J, Hoffman EA, McLennan G, Sonka M (2005) Intrathoracic airway trees: segmentation and airway morphology analysis from low-dose CT scans. Medical Imaging, IEEE Transactions on 24: Aykac D, Hoffman EA, McLennan G, Reinhardt JM (2003) Segmentation and analysis of the human airway tree from three-dimensional X-ray CT images. Medical Imaging, IEEE Transactions on 22: Cootes T (2000) An introduction to Acive Shape Models. 26. Luis Ib a nez WS, Lydia Ng, Josh Cates (2005) The ITK Software Guide-Second Edition. ITK. 27. Thomas H. Cormen CEL, Ronald L. Rivest, and Clifford Stein, editor (2001) Introduction to Algorithms. Second Edition ed: MIT Press and McGraw-Hill. Section 23.22: The algorithms of Kruskal and Prim, pp p. 28. Schneider J (1997) Rastrigin LA (1963) The convergence of the random search method in the extremal control of a many parameter system. 30. Clausen J (1999) Branch and Bound Algorithms - Principles and Examples. Copenhagen, Denmark: Department of Computer Science, University of Copenhagen. 31. Parker C (2011) Segmentation of the Airway in Lung CT Images. University of Leeds. 32. Ghuneim AG (2000) Moore-Neighbor Tracing 33. Grégoire N, & Bouillot, M. (1998) Hausdorff distance between complex polygons Sampat MP, Zhou W, Gupta S, Bovik AC, Markey MK (2009) Complex Wavelet Structural Similarity: A New Image Similarity Index. Image Processing, IEEE Transactions on 18:

52 Appendix A: Personal Reflection Being a computer student it is very difficult to work on a project which required a lot of knowledge out of your work domain. In case of this project, I have spent a lot of time on understanding the radiotherapy, CT Scans and structure of airways and lungs. I have spent a lot of time on creating the airways model, manually, which helped to make better understanding of airways in CT scan images. My first task was to understand the tools, which I was supposed to use, to get the annotations, to test the thresholds and visualize the datasets. ITK was extensively used for the segmentation of CT Images to generate a minimal spanning tree. Software documentation provided by ITK helped me a lot to understand all the filters. Code I was using for the segmentation was written by Derek Magee so I have spent some time understanding the code and making small changes to test the effects on the results. First prototype of my project didn t provide me desired results and I spent quite a lot of time to make some changes to get the desired results. Now I think that it was a very good idea to stop working on the first prototype and starting the prototype two. I really enjoyed working with Point Distribution Method (PDM). It provides me desired results and by implementing the PDM on a practical problem, I have gained a very good understanding of PDM. I tried my best to implement everything in a same application using same language but sometimes it is not good to force yourself to stick with only one programming language or a tool. That s the biggest lesson I have learnt while working on this project. Matlab is a very power full tool and it is very good to test the results of a function on Matlab (or any other available tool) before installing or integrating the libraries in your own application. Motivation plays a key role in the progress of your work. There was time when my first prototype was not working properly and I was feeling very de motivated toward the work. Once you start getting the results from your work, you put more efforts but still if you are stuck somewhere you should keep working hard. This project helped me a lot to understand the practical implication of Vision in the field of Medicine. After working on this project I want to keep working in the same field for future. 47

53 Appendix B: Resources 1- ITK Implementation for image segmentation by Derek Magee 2- Dynamic programming to fit training model in C++ by Derek Magee 3-6 Data sets of CT Scan from St James s Hospital 4- C++ implantation of Principal Component Analysis from Derek Magee 48

54 Appendix C: Minimal spanning tree of datasets After applying the connected component analysis, we select the biggest component assuming it will be tree and lungs as they are connected. While doing Thresholding, sometimes, we might break the connection between lungs and airways and this can result into a component which has only airways or airways and one of lungs otherwise both lungs and airways. Even if we lost a lung or both lungs, it is not a problem because we are only concerned with the airways. In Figure C.1 we can see the minimal spanning tree of first phase of each dataset. Figure C.1 (a) Dataset 1 Figure C.1 (b) Dataset 2 Figure C.1 (c) Dataset 3 Figure C.1 (d) Dataset 4 49

55 Figure C.1 (e) Dataset 5 Figure C.1 (f) Dataset 6 In Figure C.1 (a) we can see how both of the lungs are missing and we have a tree of airways only. In figure C.1(c) and (e) we have only one lung connected with the airways. In other cases we have both lungs connected to the airways. 50