Intelligent boundary extraction for area and volume measurement

Transcription

1 Linköping University Department of Computer Science Master thesis, 30 ECTS Datateknik 2017 LIU-IDA/LITH-EX-A--17/009--SE Intelligent boundary extraction for area and volume measurement Using LiveWire for 2D and 3D contour extraction in medical imaging Intelligent konturmatchning för area- och volymsmätning Oscar Nöjdh Supervisor : Christer Bäckström Examiner : Peter Jonsson Linköpings universitet SE Linköping ,

2 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannenslitterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet or its possible replacement for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: c Oscar Nöjdh

3 Abstract This thesis tries to answer if a semi-automatic tool can speed up the process of segmenting tumors to find the area of a slice in the tumor or the volume of the entire tumor. A few different 2D semi-automatic tools were considered. The final choice was to implement live-wire. The implemented live-wire was evaluated and improved upon with hands-on testing from developers. Two methods were found for extending live-wire to 3D bodies. The first method was to interpolate the seed points and create new contours using the new seed points. The second method was to let the user segment contours in two orthogonal projections. The intersections between those contours and planes in the third orthogonal projection were then used to create automatic contours in this third projection. Both tools were implemented and evaluated. The evaluation compared the two tools to manual segmentation on two cases posing different difficulties. Time-on-task and accuracy were measured during the evaluation. The evaluation revealed that the semi-automatic tools could indeed save the user time while maintaining acceptable (80%) accuracy. The significance of all results were analyzed using two-tailed t-tests.

4 Acknowledgments I would like to begin by thanking everyone at Sectra for the support during this thesis. Especially Daniel Forsberg, Fredrik Häll, and Jacob Bernhard whose time and interest put this thesis into a larger perspective, creating the motivation needed to complete this thesis. A very special thanks to Lukas Gillsjö, my supervisor, I could not have asked for a better person to help me find my way at Sectra. Extra thanks to those who participated in my user evaluations. Last I would like to than all my friends and family who have supported me during my time at Linköping University, I have had a great time and I could not have done it without all of you. iv

5 Contents Abstract Acknowledgments Contents List of Figures List of Tables List of Algorithms iii iv v vii viii ix 1 Introduction Motivation Aim Research Questions Delimitations Theory Color Terminology Segmentation Techniques Dijkstra s Algorithm Edge Detection D Live-Wire Turtle Map Usability Evaluation Related Work Method Pre-Study Implementation Evaluation Results Pre-Study Implementation Evaluation Discussion Results Method The Work In A Wider Context Future Work And Improvements v

6 6 Conclusion Answers To Research Questions General Conclusion Bibliography 32

7 List of Figures 2.1 Region growing process Mumford-Shah functional example Process of GrabCut Live-Wire line sticking to contour Path cooling example Example of 3D live-wire through mapping Seed point interpoliation Special cases in turtle map Example Turtle Map Samples from case Samples from case Region growing cases Smoothing comparison Live-Wire flowchart Result of the Douglas-Peucker algorithm Sobel kernels used in Canny edge detection Example results on lung tumor using 3D live-wire Results from case Results from case Mean time-on-task in case 1 and Mean accuracy and relative error in case Mean accuracy and relative error in case vii

8 List of Tables 4.1 Comments gathered from the intermediate evaluation and actions taken based on those comments Mean values of the results of case Mean values of the results of case P-values of the T-tests when comparing time-on-task P-values of the T-tests when comparing accuracy P-values of the T-tests when comparing relative error viii

9 List of Algorithms 1 Example of Dijkstra s Algorithm Relaxation Algorithm in Dijkstra s Algorithm Creating the turtle map ix

10 1 Introduction 1.1 Motivation Contour outlining in medical imaging can be a monotonous and time consuming task. Outlining the left ventricle of the heart will take a radiologist on average 15 seconds per slice [1]. A study suggests that a semi-automatic segmentation tool can speed up this process, saving valuable time [1]. When calculating volumes, if the radiologist need to draw contours in every slice containing the left ventricle, those 15 seconds per slice will quickly multiply into a much longer period of time. This thesis tries to find out if a semi-automatic tool could save time when extracting contours in two and three dimensions. 1.2 Aim The aim of this work was to create and evaluate a tool to be used when extracting 2D and 3D bodies in medical imaging. This tool was divided into two parts. First a 2D tool was created and evaluated, then two variants of 3D tools were created based on the first tool. The tools were evaluated based on user tests, measuring time and accuracy on extracted contours. The 2D and 3D tools were evaluated independently. 1.3 Research Questions The work was divided into two parts. The first part was creating and evaluating a tool used for 2D contour extraction. The first question is based on the segmentation techniques used in that tool. The tool needed to be accurate enough to be useful in medical imaging. Question 1: How can image segmentation techniques be used to create a 2D contour extraction tool suitable for medical imaging? The second part of of the work was extending the 2D tool into three dimensions. Since 3D images can be difficult to maneuver, one challenge was to make the tools interactive. The tools were based on contours extracted with a 2D tool and these contours were used to create a 3D body. The accuracy restriction was important in this case as well. Question 2: How can a 2D contour extraction tool be extended to a 3D contour extraction tool? The first two research questions discuss if such tools can be created. The third question discusses how these tools perform. User interaction was the most important factor. The 1

11 1.4. Delimitations computing performance of the tool factored into this, and it was important to create a smooth user experience. Accuracy was measured as the percentage difference between a segmented volume and the ground truth volume. The acceptable limit of 80% was chosen to represent a "general sense of the size" of the segmented body, which is what physicians are generally looking for when taking these kinds of measurements. Question 3: Can the described tools save users time compared to manual contour outlining, within acceptable (80%) accuracy? 1.4 Delimitations While several 2D segmentation techniques were researched and considered, only one was implemented and evaluated. There may be more ways to extend the 2D tool to 3D than described in this thesis. Only the two described methods were considered. Time was budgeted to improve the tools based on the feedback from the evaluation. Save that predefined time-frame, possible improvements were documented, not implemented. 2

12 2 Theory 2.1 Color Terminology This section explains how some words having to do with color and color representation in imaging are used in this thesis. The first important distinction to make is the one between color and texture. Color is a full color representation of a pixel or point in an image. This can be represented in many different ways, for example as an RGB value. Texture on the other hand refers to a gray scale value of a pixel or point in an image. This is simply represented by a number of bits where 0 represents black, and the max value represents white. Both these expressions explain the value of a pixel but with the clear distinction that one is in color and the other one is in gray scale. In medical imaging, it is rare to represent images with color. The input images in this work were all gray scale. An alpha value is an opacity value used in imaging to create transparency. An alpha value can for example be added to the RGB model to create transparent colors in images. The medical images used as input in this work did not have alpha values. If a method were to be used that required alpha values, those would have to be supplied by user interaction. This opacity value should not be confused with the alpha value used in t-statistics where alpha value represent the target probability that the null hypothesis can be discarded. T-statistics will be explained further in section section Segmentation Techniques Several segmentation techniques from the literature were considered in this thesis. This section explains the theory behind those techniques. Region Growing Region growing is an automatic, region based segmentation technique. The user selects some seed points inside the object to be segmented. Those seed points will be models for what included points should look like. Starting from one of those point, the area grows to adjacent points based on rules set from data gathered from the seed points. This process can be seen in figure 2.1. The inclusion rules are often based on color or texture value [2]. Since each 3

13 2.2. Segmentation Techniques Figure 2.1: The process of the region growing algorithm. The left image is a selected seed point and shows how it will expand to adjacent points. The image to the right show progress after a few iterations. Image from Feng [2] Figure 2.2: The process of Mumford-Shah. To the left is the original image. In the center-left image, areas of high gradient are marked in black. In the center-right image boundaries of the Mumford-Shah model. Image to the right is the segmented image. point is handled individually, the region growing technique is highly suitable for parallel programming. The union of the results from these individually handled points is the result of the algorithm. One well known implementation of this technique is the Magic Wand in Adobe Photoshop [3]. Mumford-Shah Functional The Mumford-Shah functional is a segmentation technique used to divide an image into subregions. It uses energy functions and gradient magnitudes to find similar areas to group into a sub-region [4]. Since this method does not select an area, but segment an entire image, the user interaction would be to select the area or areas that should be included in the segmented object. The process of this method can be observed in figure 2.2. GrabCut GrabCut is a graph cut based segmentation technique. The user selects an area of interest surrounding the object to be segmented. Given a trimap, the algorithm computes a "hard" segmentation in this area using graph cut. This trimap is a map of the image with only three levels of alpha values: 0 for background, 1 for foreground and a set value [0, 1] for areas around the border. Alpha values are opacity values where α = 1 is foreground and α = 0 is background. The trimap is received from user input, where the user selects areas that are foreground and background. The graph cut method is an energy optimization function based on the alpha values. The graph cut method is iterated to refine the border of the segmentation. During the hard segmentation the aplpha values can only take on three values but during the following iterations each pixel will get an alpha value α P [0, 1]. After this, the user can 4

14 2.2. Segmentation Techniques Figure 2.3: The process of GrabCut. Image from Rother et al. [3]. use a brush to select areas along the border where the initial segmentation was inaccurate. The brush can be used both to add areas that were not included in the initial segmentation, but also to remove areas that were included [3]. The GrabCut process can be observed in figure 2.3. GrowCut GrowCut is a segmentation technique that uses Cellular Automata. The user manually assign labels to some points (seed point) and the algorithm assigns these labels to all other points through automa evolution [5]. This method is essentially a variation of region growing that uses cellular automata. Live-Wire Live-Wire is a semi-automatic segmentation technique based on Dijkstra s shortest path algorithm. Dijkstra s algorithm is used by defining a graph G(E, V) where the vertices are the pixels (points) in the image, and edges the edges connect each pixel to its surrounding (neighbouring) pixels. The user only needs a few clicks to draw contours around an object in an image. The image is preprocessed and each pixel is assigned a weight based on how likely it is that that pixel is on an edge. A point on the edge of the object will be cheap, making it preferred to be traversed in Dijkstra s algorithm. The edge weights used by Dijkstra s algorithm are defined using these pixel values together with difference in texture values of the points the edge connects. When the user clicks on the image, a starting point (seed point) is placed. Dijkstra s algorithm is run calculating the shortest path from the seed point to every other point. When the user moves the mouse the shortest path from the seed point to the current location is displayed. Another click freezes the current path and places another seed point. This is continued until two lines crosses each other and the area is complete. Figure 2.4 show how the line stuck to the edge when the mouse pointer was moved along the dashed line. One famous implementation of live-wire, called Intelligent Scissors, was published in Intelligent Scissors uses Laplacian zero-crossings, gradient magnitudes, gradient di- 5

15 2.3. Dijkstra s Algorithm Figure 2.4: Line sticking to contour in live-wire. The dashed line show the mouse s movement. rections and pixel values (colors) to find the edges in the image [6]. This algorithm was implemented in Adobe Photoshop under the name Magnetic Lasso. 2.3 Dijkstra s Algorithm Dijkstra s algorithm is a single-source shortest path algorithm developed by Edsger Dijkstra in The result of the algorithm is the shortest paths from the starting vertex to all other vertices. The algorithm maintains a set S of all visited vertices, it requires a graph G(V, E) and a weight function w. Each iteration chooses a vertex u P V S with the shortest path in the set, and relaxes that vertex. The algorithm is finished when V S = H. Dijkstars algorithm has a time complexity of O(V 2 ). If the graph is very sparse, E = o(v 2 /lg(v)), the algorithm can be improved to run in O((V + E) lg(v)) by using a min-heap to locate u P V S with the shortest path. If all vertices are reachable from the seed point, O((V + E) lg(v)) = O(V lg(v)) [7]. An example implementation can be found in algorithm 1. The relax method can be found in algorithm Edge Detection There are many edge detection methods. This section describes some of the most commonly used in live-wire algorithms. Laplacian Zero-Crossing One common way of finding edges in live-wire is marking Laplacian zero-crossings [6, 8]. Detection of zero-crossings is divided into three main steps: 6

16 2.4. Edge Detection Algorithm 1 Example of Dijkstra s Algorithm 1: function DIJKSTRA(G, w, s) 2: Initialize-Single-Source(G, s) 3: S Ð H 4: Q Ð G.V 5: while Q H do 6: u Ð ExtractMin(L) 7: S Ð S Y tuu 8: for each vertex v P G.Adj[u] do 9: Relax(u, v, w) 10: end for 11: end while 12: end function Algorithm 2 Relaxation Algorithm in Dijkstra s Algorithm 1: function RELAX(u, v, w) 2: if u.d + w(u, v) ă v.d then Ź v.d is the weight of the currently shortest path to v 3: v.d Ð u.d + w(u, v) 4: v.π Ð u Ź v.π is the predecessor of v in the shortest path to v 5: end if 6: end function 1. Convolve the image with a directional second derivative of a Gaussian filter. These filters are also known as Laplacian of Gaussian filter. 2. Localize the zero-crossings by finding neighbouring pixels with different signs (+/-). 3. Check the alignment and orientation of the zero-crossings. Since the third step is relatively expensive and there is little gain, it can be excluded [9]. The found zero-crossings will be on potential edges. A smoothing (Gaussian) filter can be applied to the image before filtering for zero-crossings, to lower the noise level. A lower noise level will reduce the amount of false-positives when looking for edges, demonstrated in figure 4.2. Gradients The image gradients can be used to find edges. A strong gradient (i.e. a gradient with high magnitude) indicates a probable edge. Since contours rarely have sharp turns, the gradient direction can be compared to the direction between two neighbouring points. Neighbouring points in images are defined as the eight pixels surrounding each pixel. High differences between pixel directions and the normal of gradient directions can be made expensive to filter out sharp turns [6]. Canny Edge Detection Some newer implementations of live-wire has used Canny edge detection to find the edges in the images. This method is more suitable for medical images because it is less sensitive to image noise [10]. It was presented by John Canny in 1986 and is another widely used edge detection technique [11]. It is a relatively simple algorithm that can be divided into five steps [12]. 1. Convolve the image with a gaussian filter. 2. Use an edge detection operator (e.g. a Sobel filter) to find the image gradients. 7

17 2.5. 3D Live-Wire 3. Apply non-maximum suppression to thin lines. 4. Apply double threshold to filter out very weak edges and set strong edges. 5. Track edges to remove weak edges not connected to a strong edge. Like in Laplacian zero-crossings the first step is used to suppress noise in the image. A Sobel filter is then applied to the image to find the gradients in the image. These gradients consist of a magnitude and a direction. The gradients will have a high magnitude on the edges with a direction from low to high pixel values. In the third step, each pixel s gradient magnitude will be compared to its two neighbours in positive and negative direction of its gradient. If it is not the highest of the three it will be suppressed. This thins the lines to make sure that only the true edge is left. A double threshold is applied to the gradient magnitudes. Gradients with a magnitude lower than the lower threshold will be suppressed. Gradients with magnitudes between the thresholds will be considered weak edges. If the magnitude is above the higher threshold it will be considered a strong edge. In the fourth step all weak edges that are not connected to a strong edge will be suppressed. This removes leftover noise. This is done by promoting all weak edges to strong edges if and only if they are neighbours to a strong edge. This is iterated so that chains of weak edges connected to a strong edge will be strong edges. All points that are still weak edges when this iterative process is complete will be suppressed [12]. Path Cooling Some implementations of live-wire uses path cooling to minimize the number of clicks required by the user [6]. If two shortest paths share a common point, they also share the path between that that common point and the seed point. This is demonstrated in figure 2.5. When a point is included in a lot of user-created paths or have been included for a set time, it is probable that the user sees it as correct. An automatic seed point can then be created at that point. In figure 2.5, the paths that are marked in red and blue will be "frozen" and will not change when the user moves the mouse, even if a shorter path is found. On simple contours like a circle this technique makes it possible to extract the contour without having to place any manual seed points except for the first one. Training Segments On-the-fly training can be used in live-wire to make pixels with similar properties as the ones selected by the user cheaper. These properties can include color/texture value or color of the pixels in the direction of the gradient. Segments that the user have locked in are used as a template for how what properties other contour pixels should have. Similar pixels can be made cheap when calculating the shortest paths [6] D Live-Wire The basic idea of 3D live-wire is to use a few user made contours to find the contours of a 3D object. One way of doing this is described by literature on the subject, a second way is presented in this thesis. The first method was described by Poon et al. [13] in The idea is to map the seed points of user created contours to another projection and create new contours there [13]. The method created for this thesis uses interpolation of seed points to create contours in one projection only. 3D Live-Wire Through Mapping A first way of implementing 3D live-wire was described by Poon et al. [13]. The idea is to let the user create manual 2D live-wire contours from two orthogonal angles and then auto 8

18 2.5. 3D Live-Wire Figure 2.5: Seed point is marked in red. The two paths have the blue part in common in their shortest path to the seed point. segment several contours from the third orthogonal angle. Each orthogonal plane that has intersections with the manually created contours will be automatically segmented, creating contours in several parallel planes. These intersections are used as seed points for the automatically segmented contours. This is shown in figure 2.6. The intersections (seed points) have to be ordered in the new plane to be connected as a contour. For this purpose an algorithm based on L-systems turtle is used [13]. Figure 2.6: The red and blue contours are manually segmented. The green contour is automatically segmented using seed points where the other contours intersect with the green plane. The image on the right shows the plane where the green contour was segmented, with intersections marked with white circles. Several planes parallel to the green plane will get automatically segmented with this method to create a 3D body. Image from Poon [13] 9

19 2.6. Turtle Map Figure 2.7: Seed points are blue, manually segmented (2D live-wire) contours are red, automatically segmented contours are green. The seed points from the manually segmented contours are adapted to the contour in the next slice then connected with live-wire techniques. 3D Live-Wire Through Interpolation The second way of extending live-wire to 3D is to let the user create several 2D live-wires with some slices between. The seed points from these contours can be interpolated to the other slices and there connected by using the same techniques as 2D live-wire. This process is demonstrated in figure 2.7. One notable disadvantage with this method is that the adaption of the seed points is imperfect. This means that the interpolated seed points may, especially on irregular shapes, not be places on the contour in the next slice. This error could potentially accumulate to a substantial error in the segmented volume. This method is faster than manually placing contours due to a number of reasons. One reason is that the user does not have to manually place more seed points. This removes the reaction time from the user, which is generally the slowest part of the live-wire algorithm. Another reason is the nature of Dijkstra s algorithm. In 2D live-wire, Dijkstra s algorithm will, on every click, find the shortest path from the clicked on point to every other point. When implemented with a min-heap, this is an O(V/lg(V)) operation, where V is the number of vertices [7]. When connecting the interpolated seed points, however, Dijkstra s algorithm can be interrupted when the next seed point is visited. This will not reduce the worst case time of the algorithm but since the seed points should be placed on a contour they will be visited early in the process. 2.6 Turtle Map The points where the plane of an automatically segmented contour intersects with the existing contours will be used as seed points in mapping based 3D live-wire. When found, these points will be unordered in the new plane. L-System s turtle is an algorithm that can be used in 3D live-wire to order the seed points in the automatically segmented contours. The algorithm is essentially a depth-first search that orders the points in clockwise order in the new plane [13]. The new plane will intersect with each manually segmented contour in an even amount of places. These points are paired with the opposing point from the same contour. Each pair are then placed on the turtle map with a straight "road" drawn between them. This is done for all intersecting contours. The algorithm then traverses the map by placing a "turtle" on a staring point. The turtle follows the road straight, turning left when it reaches an intersection and turning 180 degrees when reaching another point. The algorithm is finished when the turtle has returned to the staring point. The order of the points will be the order in witch the turtle visited them on the map. A pseudo implementation of this can be found in algorithm 3 and figure 2.9 shows and example map and how it would be traversed. There are special cases 10

20 2.7. Usability Evaluation Figure 2.8: Special cases possible in turtle-map. Threes represents seed points, ones represents roads. Note that in all three cases, seed points have been placed where an intersection would have been. to the turtle map algorithm not handled in algorithm 3, some of these are displayed in figure 2.8. Note that such cases appear when a seed point happen to be placed on an intersection. Algorithm 3 Creating the turtle map 1: function TURLTE(P) 2: M Ð 0 Ź M is the map, starts out empty 3: for each pair p in P do 4: M[p.point1] Ð 3 5: M[p.point2] Ð 3 6: for each point r between p.point1 and p.point2 do 7: M[r] Ð M[r] + 1 8: end for 9: end for 10: TraverseMap(M) 11: end function 2.7 Usability Evaluation Usability evaluation can be used to improve a system under development or to evaluate a complete system. User tests should normally be performed by users with similar background and experience as end users [14]. During these tests, subjects are exposed to a series of tasks. Users should think out loud while performing these tasks, so that thoughts and impressions can be gathered. Metrics can be collected to evaluate the system or compare versions of the same system. Sample Size Studies have shown that five participants is sufficient to uncover about 80% of issues in a system and that these are usually the most severe issues [15]. When collecting metrics, more is usually better. Sauro argues in his article [16] that a small sample size is fine, as long as significance is analyzed using t-statistics. Two-tailed t-statistics take sample size into account and give a probability measurement of how probable it is that a difference can be found between to groups of samples (e.g. a group of test samples and a group of control samples). A null hypothesis is presented to the test, usually stating that no difference in mean values can be found when comparing the two groups of samples. The test will then return a probability value stating how probable it is 11

21 2.8. Related Work Figure 2.9: Example turtle map with arrows showing traversal. Ones represent road, twos represent intersection, threes are seed points. The green border marks the staring/ending point. The small numbers by the arrows are the order of traversal. that this null hypothesis can be thrown out. If a higher number of samples are used, the more certain the returned probability value will be. Time-On-Task Time-on-task is a widely used usability metric. It is simply a measurement of how long it takes a user to complete a specified task. This metric has been used since the nineteenth century and is reliable and well used [17]. When measuring time-on-task in smaller sample sizes (ă 30) t-statistics should be used to analyze the result because it takes the sample size into account [16]. 2.8 Related Work The 2D tool in this thesis is based on Inteligent Scissors presented by Mortensen and Barett [6] as well as the techniques presented by Kimmel and Bruckstein [18]. The 3D tools are mainly inspired by the work done by Poon et al. [13]. 12

22 3 Method This chapter describes the method that was used in this work. The method was divided into three parts: pre-study, implementation, and evaluation. 3.1 Pre-Study Since the goal of this work was to implement and evaluate one segmentation tool, the goal of the pre-study was to research existing tools and choose one to implement. Tools were initially evaluated based on a few requirements that Sectra provided as goals for this thesis. The tool should: 1. Be fast 2. Require little interaction from the user 3. Be accurate 4. Work well on well defined lesions and objects in CT/MR data Speed and accuracy are well defined metrics used in most literature on this topic. Because those metrics were well defined, the corresponding requirements did not have to be defined further. The amount of interaction required from the user was estimated as mouse button presses ("clicks") since this is something that medical IT wants to limit [19]. To find out if a tool or technique was well suited for CT/MR images, earlier applications were studied and considered. 3.2 Implementation This section describe the implementation process of the semi-automatic tools. Algorithm Input Input to the algorithm consisted of two things: texture values that represented the image pixels, and window levels. The texture values were arrays of 16 bit values where 0 represented 13

23 3.3. Evaluation black and the max value represented white. The window levels were a double threshold set by the user. The texture values in the range [lower threshold, upper threshold] would be displayed in the the black/white spectrum. All values lower than the lower threshold would be displayed as black, and all values over the upper threshold would be displayed as white. This double threshold made it possible for the user to highlight different tissues and textures in the body. 2D Semi-Automatic Segmentation Once a suitable tool was selected, that tool was implemented on a branch of Sectra s PACS (Picture Archiving and Communication System). The system is an image handling and viewing system, created for radiology and pathology. Implementing directly into Sectra s system gave the implementation a good framework of already implemented elements. Image rendering, mouse and keyboard handling, and drawing tools were all already in this framework. Sectra s system had a tool where the user could place points using the mouse pointer, and straight lines would be drawn between these points. This tool served as the foundation for the new tools. All code was written in C#. The system had a testing environment in place that was used during implementation of these tools as well. 3D Semi-Automatic Segmentation The next step was to extend the 2D tool to three dimensions. Both of the considered methods were implemented and evaluated. Since the 3D tools were implemented with a coordinate system where 1 unit equaled 1 millimeter, contours were auto-segmented with one millimeter apart. To calculate the volume of the segmented object, the areas of all the contours were summed: V = ÿ n=1..c 1 D n,n 1 (A n + A n 1 )/2 (3.1) where c is the number of contours, V is the volume, A n is the area of contour n, and D n,n 1 is the distance between contours n and n 1. As mentioned D n,n 1 was usually one millimeter, but the user could manually delete contours to create a wider gap between two contours. Since the distances between contours were so small, this linear method was used rather than a more advanced method like spline interpolation. 3.3 Evaluation Intermediate Evaluation The 2D tool was evaluated by a small group of users before completion and was then improved based on the feedback from that evaluation. During this intermediate evaluation 10 developers at Sectra got to try the tool. First, the users got the chance to familiarize themselves with the tool to minimize the effect of lack of experience. After the users were familiar with the tool, time was recorded while the users segmented three different shapes. The users used both the semi-automatic tool and a manual drawing tool to segment the shapes. Time and accuracy of these segmentations where compared. The purpose of this evaluation was mainly to give the users a sense of meaning while their comments on the tool where recorded. The comments was used to improve the 2D semi-automatic tool. Final evaluation Each tool was evaluated in four steps: 1. A short explanation of the theory behind the tool. 14

24 3.3. Evaluation 2. Let the user familiarize themselves with the tool. 3. Let the user segment case Let the user segment case 2. All tools, including the drawing tool were evaluated using the same procedure. Due to lack of time, some subjects skipped the final step. Both metrics presented in this section were collected on every segmented case. Metrics During the evaluation, two metrics were used: time-on-task and the accuracy of the calculated volume. Time-on-task was measured form when the tool was selected in the drop-down menu to when the user said that they were satisfied with the result. The reason that the user was let to chose the end time was because the auto-segmented volumes could be interactively improved by the user and that time/improved accuracy should be considered in the result. The metric Z T was calculated as the mean time-on-task of one tool applied on one case. The result of the time-on-task metric was analyzed using t-statistics because of the limited (ă 20) sample size [16]. Since the purpose of this thesis was to save time when measuring volumes, this was a natural choice of metric when evaluating the segmentation tools. Accuracy was calculated as Z A = V seg /V true where V seg is the segmented volume and V true is the ground truth volume. The ground truth volume was extracted using manual segmentation in very slow and precise sessions. Acceptable accuracy was chosen as 80%- 120% (0.8 ď Z A ď 1.2), which is higher than some similar studies on medical images [20]. Accuracy was also presented as relative error, calculated as Z E = Z A 1. Making Z E ď 0, 2 an acceptable score. The reason that two metrics based on accuracy were presented, was because they both presented some interesting properties. While relative error gave the magnitude of the accuracy, the accuracy measurement described if the segmented volume was bigger or smaller than the ground truth. Sample Size Because of the lengthy user evaluations combined with a limited time budget, a small (ă 20) sample size was used. Using a small sample size can still give reliable results if the results are analyzed using t-statistics since t-statistics takes sample size into account [16]. Two-tailed t-tests were performed comparing the accuracy and time-on-task results between implemented tools and manual segmentation, as well as between the two implemented tools. The alpha value of these tests was chosen as 0.05 as this is common practice in clinical studies. Cases All tools were evaluated on two cases. The first case was one that the tools were known to handle well, the second case was more difficult. The first case was a lung tumor that ranged through 10 images. Some samples from this case can be seen in figure 3.1. Due to the black background and clear contours this was an easy case that the semi-automatic tools should handle well. The tumor in this case split into 15

25 3.3. Evaluation two towards the top images (seen to the right in figure 3.1). A tool that did not handle split cases would therefore have a lower accuracy in this case. Figure 3.1: Samples from case 1, a lung tumor. In the rightmost image, one can see how the tumor split into two figures. The second case was a brain tumor that ranged through 42 images. Some samples from this case can be seen in figure 3.2. This tumor was varying in color and shape through the images. The color of the tumor in a single image was often speckled. These properties made this second case much tougher than the first one. This tumor never split into two, meaning that the tools would not have to handle this to get a perfect accuracy rating. Figure 3.2: Samples from case 2, a brain tumor. As shown, this case was very varying in shape and color. 16

26 4 Results This chapter presents the results from the process described in the method chapter. First the results from the pre-study will be presented as a list of possible techniques with pros and cons and what technique was chosen for this thesis. Secondly the implementation of those techniques will be presented. This section will include technical choices made during implementation. Thirdly the result of the evaluations, both the intermediate and the final evaluation, will be presented. 4.1 Pre-Study Several techniques were researched and considered according to the requirements described in the method chapter. The techniques are described below with pros and cons why that technique was suitable or not. Region Growing Because of its suitability for parallel execution, region growing can be made very fast. This technique also requires very little input from the user as only the initial seed points have to be manually placed. The accuracy of this algorithm depends heavily on the object to be segmented. A single colored object with a distinct edge that is on a different colored background will be segmented very accurately. In some applications on CT and MR images, the textures in the object are very similar to the textures of the background and this makes region growing unsuitable for some medical imaging. The examples in figure 4.1 are captured from an earlier implementation in Sectra s system. As can be seen in the figure, this implementation did not handle the more difficult brain tumor case very well. While a good candidate, the fact that a 3D version of this technique was already implemented and tested by Sectra, ruled it out for this thesis. Mumford-Shah Functional According to the work done by Chen, Yang and Cohen [21] the method proposed in Mumford [4] is slow and not suitable for medical imaging. With the changes they propose however, 17

27 4.1. Pre-Study Figure 4.1: Region growing works well in the image to the left (a lung tumor), not very well on the white object in the image to the right (a brain tumor). it was a valid candidate. The changes proposed by Chen, Yang and Cohen [21] made the technique faster and more accurate in medical imaging. It requires little user interaction. GrabCut According to Rother [3], GrabCut is a very fast technique. Test images required less than a second for initial segmentation on an image of 450x300 pixels and then 0.12 seconds per brush mark from the user. It is however not accurate enough on medical imaging, as shown in Kalshetti [22]. GrowCut GrowCut has a speed comparable to GrabCut, making it fast enough for this thesis. The algorithm only requires a few brush marks from the user to give an accurate result. Although the authors of the paper presenting this technique have a few examples from medical images, it has not been proven to work well on high detail medical images [5]. Live-Wire Live-wire is slower and require more user interaction than all of the above mentioned techniques [23, 3]. What it does do that the others don t, is giving the user instant feedback. While the other techniques are automatic after the user initiates the segmentation process, live-wire tells the user what the result of their next click will be and let the user decide if that is satisfactory. Live-wire is accurate and have been proven to work well in medical imaging [6, 24, 23]. Final Choice The final choice was to use live-wire. This was because of the interactivity the instant feedback brings. Because medical imaging present very differing scenarios that the algorithm needs to handle, giving the user an idea beforehand what the result will be helps to handle a wider array of cases. The result of a process that the user only initializes can be difficult to 18

28 4.2. Implementation Figure 4.2: Left is original image. Middle is zero-crossings found without applying a smoothing filter first. Right is zero-crossings found when applying smoothing filter first. anticipate. Another deciding point was that the contour nature of the result from live-wire is very suitable for extending this technique to three dimensions. 4.2 Implementation 2D Live-Wire The implemented Live-Wire initially only used Laplacian zero-crossings to detect edges and assign weights since it is a fast and accurate way to detect edges. To limit the influence of the noise, a Gaussian filter was applied to blurr the image before using the zero-crossings technique. This removes some of the noise but also some detail. The result of applying zero-crossing with and without blurring the image can be seen in figure 4.2. The Laplacian zero-crossings technique is by itself not suitable for medical imaging because of the high noise levels in those images [24]. Because of those limitations, Canny edge detection was added when calculating the weights. Path cooling was used to limit the number of clicks required from the user. When a segment was more than 30 pixels long and had been included in 80% of the calculated paths since the last seed point, the segment from that point back to the seed point would be locked in the same way as if the user had placed a seed point there. A locked segment means that the segment will not be changed even if the user moves the mouse. A "trained" factor was added to the weight calculations. This factor compares all pixel values to the ones of the locked segments. Points with similar values are made cheaper and distant values are made more expensive. The pixel values in the direction of the gradients are also compared to try and keep the same value in the direction of the gradient at all times. This is because segmented objects are usually uniform in color. 19

29 4.2. Implementation Figure 4.3: A flowchart describing the process of the implemented Live-Wire. A flowchart of the algorithm can be found in figure 4.3. To make the resulting area easier to interact with, it was simplified using the Douglas Peucker algorithm [25]. This algorithm reduces the number of points required to represent each contour, giving the user fewer points to move if the result was unsatisfactory. The Douglas-Peucker algorithm takes a starting point and an ending point (in this case two seed points) and all the ordered point between these. A line is drawn between the starting and ending point, and it is calculated what intermediate point is furthest from this line. If the "worst" point is further away than a set value ɛ that point has to be included to keep the precision of the curve. If it is closer than that it can be removed and the line between the staring point and ending point will be an appropriate approximation of the curve. If the point should be included, the algorithm is recursively called with the subsets [starting point, worst point] and [worst point, ending point]. The result is a subset of the original points. An example result of the Douglas-Pucker algorithm can be found in figure 4.4. The user could move points by dragging them. Lines were also draggable, moving two points on either side of the line. The entire contour can be moved by dragging a point placed in the middle of the area. Kernels And Weights The Laplacian of Gaussian kernel used in the implementation was a 7x7 kernel with standard deviation 2.4. The Gaussian kernel used in blurring the image was a 5x5 kernel with standard deviation 1.4. A Sobel filter was used to find the gradients of the image, the kernels used in that convolution can be found in figure 4.5. Weights were calculated according to this formula: ( Z L ) + (1 + 9 Z C ) + 20 Z T, where Z L and Z C are 0 if an edge was found and 1 if one was not found, using Laplacian 20

30 4.2. Implementation Figure 4.4: The image to the left is the segmented contour before applying the Douglas-Pucker algorithm. To the right is the result of the algorithm Figure 4.5: Sobel kernels used in Canny edge detection. X and Y direction kernels respectively. zero-crossings and Canny edge detection respectively. The weighting factors 19, 9 and 20 were chosen based on extensive testing on different cases and have no other proof why just they were chosen. It could be possible to find better weighting constants, but these were good enough and no better ones could be found during testing. Z T is the training factor, a number between 0 and 1 described in section 4.2. The reason that the weights were chosen with an offset of 1 was so that no pixels would be "free" to traverse. This means that no "fake" edges will be followed far out of the way of the real contour. It also, however, means that some small shapes will be missed, since taking shortcuts may be cheaper than strictly following the contour. Intermediate Evaluation A smaller evaluation was performed towards the end of the 2D live-wire development. While some metrics were gathered, t-tests on these showed that no significant conclusions could be made with respect to time-on-task and accuracy. The main purpose of this evaluation was to gather comments and improvement ideas on the 2D live-wire tool. The comments gathered during the intermediate evaluation, and the actions taken to improve based on them, can be found in table

31 4.2. Implementation Comment Long loading times Sometimes difficult to know when loading is finished Users will follow contour with mouse even if they do not have to Expect seed-point on contour no matter where the click is Some areas of objects are easier to segment with manual segmentation Too many clicks To many points to move when not happy with the result To few movable points to be able to get the right contour when not happy with the result Implemented/Possible solution Implemented threaded convolution Improvements were made to the mouse icon updates Possible to make points passed over by mouse cheaper, this was not implemented however Possibly because users were unaccustomed to live-wire. Nothing was implemented Added functionality to hold down mouse button and manually segment parts of objects Path cooling was implemented as an answer to this comment Possible solution would be to lower the sensitivity of the Douglas-Pucker algorithm Possible solution is to add ability to add point on contour by holding a key down and clicking Table 4.1: Comments gathered from the intermediate evaluation and actions taken based on those comments. Figure 4.6: Example results from using auto-wire. The images to the far left and right were manually segmented in an added time of 15 seconds. The middle contours were auto segmented with auto-wire in a total time of 3.5 seconds. 3D Auto-Wire Auto-wire, as the 3D live-wire tool was referred to, was implemented using the interpolated seed point method described in chapter 2.5. Auto-wire works in five steps: 1. User segments two contours with 2D live-wire. 2. From the two contours, each seed point creates a new seed point in the next slice. This is done by creating a seed point 1/-1 unit in the direction of the vector normal to the viewed plane. 3. All new seed points are moved to the cheapest point in a small surrounding area. 4. The new seed points are connected using the same techniques as in 2D live-wire. 5. Steps 2-4 are repeated until every slice between the manually segmented contours have been processed. Step 1-4 are summarized in figure 2.7 Note that while the user always begin with two manual live-wire segmentations, these do not have to be on the very top and bottom ones of the tumor. These two can be a few slices apart. Adding a third manual live-wire segmentation a few slices away from the first 22