MULTIPHASE LEVEL SET EVOLUTION WITH APPLICATIONS TO AUTOMATIC GENERATIONAL TRACKING OF CELL DIVISION OF ESCHERICHIA COLI. A Thesis.

Transcription

1 MULTIPHASE LEVEL SET EVOLUTION WITH APPLICATIONS TO AUTOMATIC GENERATIONAL TRACKING OF CELL DIVISION OF ESCHERICHIA COLI A Thesis Presented to the Faculty of San Diego State University In Partial Fulfillment of the Requirements for the Degree Master of Science in Applied Mathematics by Erin L. Daly Spring 2012

2

3 iii Copyright c 2012 by Erin L. Daly

4 iv ABSTRACT OF THE THESIS Multiphase Level Set Evolution with Applications to Automatic Generational Tracking of Cell Division of Escherichia coli by Erin L. Daly Master of Science in Applied Mathematics San Diego State University, 2012 Segmentation and generational tracking of cells in microscopy video is a growing area of research in the biomedical field. A universal automated method to detect single cells and construct frame to frame lineage trees has yet to be developed. The success of a generational tracking method relies on the accuracy of the segmentation algorithm. Individual cells must be detected in each frame before tracking and frame to frame object correlation can be conducted. Active contour based algorithms have proven to be very suitable for detection of different cell types with varying image conditions. However, traditional active contour methods can be highly inaccurate when segmenting cluttered images and are computationally expensive. To address these concerns, we explore a non-gradient based level set implementation, which is successful in detecting objects in low quality, congested images. In this paper, we explore various versions of an active contour, level set, method for image segmentation. We present an advanced multiphase level set method for image segmentation with new approaches for algorithm initialization, termination, and identification of individual cells. Specifically, we focus on the detection of individual Escherichia coli (E. coli) cells undergoing cell division in individual frames of phase-contrast microscopy video.

5 v TABLE OF CONTENTS PAGE ABSTRACT... iv LIST OF FIGURES... vii ACKNOWLEDGMENTS... viii CHAPTER 1 INTRODUCTION APPLICATION BACKGROUND Escherichia coli Generational Tracking of E. coli Image Segmentation METHOD BACKGROUND General Level Set Method Weaknesses of General Method Mumford-Shah Segmentation Variational Level Set Method METHOD Multiphase Level Set Four Phase Level Set Method Initialization Stopping Criterion Connected Components Labeling RESULTS Ground Truth Results: Test Data Processing Conclusions of Initial Testing Bacterial Growth Results Simple Detection: Small Population of Cells Complex Detection: Large Population of Cells CONCLUSION Potential for Basic Generational Tracking... 36

6 vi 6.2 Existing Cell Tracking Methods BIBLIOGRAPHY APPENDICES A FAILED METHODS B MORE INFORMATION ON EQUATIONS C METHOD FLOW CHART... 51

7 vii LIST OF FIGURES PAGE Figure 2.1. High resolution images of E. coli cells... 4 Figure 2.2. Hypothetical lineage tree of one E. coli cell... 5 Figure 3.1. Level set method illustration... 8 Figure 4.1. Partition of the image domain into four regions using two level set functions. 16 Figure 4.2. Level set initialization types Figure 4.3. Connected components labeling method Figure 5.1. Evolving contours overlaid on original test image Figure 5.2. Test image 1 results Figure 5.3. Evolving contours overlaid on original test image Figure 5.4. Test image 2 results Figure 5.5. Initial video frames Figure 5.6. Evolving contours overlaid on original frame Figure 5.7. Frame 1 results Figure 5.8. Evolving contours overlaid on original frame Figure 5.9. Frame 6 results Figure Complex video frames Figure Evolving contours overlaid on original frame Figure Frame 29 results Figure Evolving contours overlaid on original frame Figure Frame 62 results Figure Frame 62 with superimposed initial contours Figure Frame 62 binary output Figure A.1. Evolution of variational level set function on frame 1 of E. coli video... 43

8 viii ACKNOWLEDGMENTS I would like to thank my adviser, Peter Blomgren, whose guidance and support enabled me to explore various approaches to the selected problem and develop a detailed understanding of the subject. I would also like to thank my thesis committee members. It is an honor for me to have Paul Paolini and Joseph Mahaffy serving on my thesis committee, providing their input and support to complete this thesis.

9 1 CHAPTER 1 INTRODUCTION The study of cell movement, segmentation, and lineage (or generational tracking) is a critical research area in the biomedical field. Time lapse movies, or microscopy video data, are becoming increasingly popular in biomedical research as they allow time courses of cell division, gene expression, and DNA transfer at the single cell level. However, extracting data from video on the single cell level is a major challenge. Traditionally, the most accurate and reliable cell detection methods are manual in nature. These methods are impractical for large microscopy video data sets with rapid cell division and an exponentially growing number of cells. There is a need for a robust, automated method to improve data processing capability and remove the operator error present in manual segmentation of large cell populations. The success of any such automated generational cell tracking method relies on the accuracy of the segmentation algorithm applied to each video frame. Many image segmentation techniques have been employed for the detection of objects in imagery, including the watershed transform [19] and Canny edge detector [16]. These traditional gradient dependent methods fail to segment individual cells whose boundaries are not uniquely defined by the gradient, which is often the case for individual cells in microscopy video. Active contour models [22] have proven to be very suitable for detection of objects under varying image conditions. The active contour model is a variational framework for extracting object boundaries in an image by evolving a contour subject to constraints of the image. The curve is evolved according to the Euler-Lagrange equation whose solution minimizes an appropriate energy functional. The boundary of the piecewise-constant object region in the image is defined by the minimum of the energy functional. The level set method [13] can be constructed as a variational active contour problem in which the solution to the system of PDEs governing the level set evolution minimizes an energy functional, similar to the Mumford-Shah functional [3]. The evolving curve can be defined as a level set function φ such that the zero level set φ = 0 defines the object boundary. Evolution of a single level set function φ can only model two distinct image regions (or phases). The single level set function shows weakness when applied to images with a large cell population clustered together as one cell mass. For these more complicated situations, additional level set functions are considered to deal with more complex topology. The multiphase level set method [20] can model n phases using m = log 2 n level set functions.

10 2 The multiphase level set method for image segmentation and individual cell detection is the focus of this paper. We present an advanced multiphase level set method for image segmentation with new approaches for algorithm initialization, termination, and identification of individual cells. Our method accurately detects the boundaries of individual cells using a multiphase level set and separates the homogeneous populations of cells to distinctly label each individual cell using connected components labeling. The final multiphase level set method is applied to successive frames of an E. coli cell division phase-contrast microscopy video. E. coli is one of the most diverse bacterial strains with the ability to transfer DNA horizontally through an existing population. As such, new strains of E. coli evolve easily and are of particular interest in the biomedical field. Automatically detecting E. coli cells in microscopy video and constructing a reliable frame to frame lineage tree for each cell would allow detailed data analysis of the evolution and growth of E. coli strains. We show that automatic detection of individual E. coli cells can be conducted in large, congested populations. More importantly, we show that our method can separate distinct cells undergoing cell division regardless of the direction or size that they grow. The cell segmentation results of each frame can be compared to construct frame to frame generational lineage trees for each cell using a combination of existing techniques.

11 3 CHAPTER 2 APPLICATION BACKGROUND The focus of our cell detection and tracking effort is time lapse microscopy video of cell division of E. coli. Escherichia coli (E. coli) is one of the most diverse bacterial species, with only twenty percent of the genome common in all strains [19]. This bacteria grows quickly and has the ability to transfer DNA horizontally through an existing population. As such, new strains of E. coli evolve easily. The study of cell movement, segmentation, and lineage has been an important research area in the biomedical field with a focus on tracking the growth and evolution of E. coli strains. Specific attention is being paid to cell segmentation (object detection) and tracking in phase-contrast microscopy video. 2.1 ESCHERICHIA COLI Prokaryotes, such as bacteria, lack a membrane-bound nucleus and have no membrane organelles. Therefore, detection of such cells requires accurate identification of individual cell walls, often in congested populations with possible collision or overlap, as opposed to the detection of distinct, clearly separated cell nuclei. Escherichia coli (E. coli) is rod-shaped bacterium that commonly inhabits the intestinal tract of warm-blooded animals. Most strains of E. coli are harmless and serve a useful function in the host by producing vitamins and suppressing the growth of harmful bacteria. However, there are pathogenic strains of the bacterium, which cause illness or disease [17]. As such, the growth and automatic generational tracking of E. coli bacteria is of particular interest. Typically, prokaryote cell division occurs through binary fission. In this method of cell division, the original DNA is replicated and one cell gives rise to two identical daughter cells. The resulting DNA molecules attach to different parts of the cell membrane and the cell begins to pull apart (grow). A septum forms at the center of a growing E. coli cell through inward growth from opposing sides of the outer cell membrane and cell wall. This septum eventually pinches off producing two descendant cells. Figure 2.1 illustrates the appearance of E. coli cells during the process of cell division [8]. These high resolution images are provided to give a clear illustration of the cell division process and do not resemble the low quality microscopy images to be processed. In phase-contrast microscopy video, E. coli cell soma in the focal plane appear as a dark region surrounded by a white halo.

12 4 Figure 2.1. High resolution images of E. coli cells. Provided to give a clear illustration of the cell division process and do not represent the appearance of E. coli cells in low quality phase-contrast microscopy video to be processed for this application. (a) Collection of E. coli cells during cell division. (b) Single E. coli cell during cell division. Source: D. KUNKEL, Cell division: Binary fission and mitosis. accessed January 2012, May GENERATIONAL TRACKING OF E. COLI The study of cell movement, segmentation, and lineage has been an important research area in the biomedical field with a focus on tracking the growth and evolution of E. coli strains. The goal of any method (manual or automated) is to create a generational track of each cell from frame to frame, similar to that illustrated in Figure 2.2 [21]. This lineage tree would be constructed at the cell level to relate old cells and new cells, formed by cell division, from the frame at time t to those in the frame at time t + 1. Constructing a reliable lineage tree for each cell would allow detailed data analysis of the evolution and growth of E. coli strains. Time lapse movies are becoming increasingly popular in biomedical research as they allow time courses of gene expression dynamics to be established at the single cell level, [21, p. 102] especially those with high temporal and spatial resolution. Extracting data on the single cell level from video is a major challenge facing the biomedical field as the amount of time lapse microscopy video available continues to grow. Traditionally, the most accurate and reliable tracking methods are manual in nature, with limited computer automation. These methods are impractical for data sets with rapid cell division and an exponentially growing number of cells [23]. According to Dzubachyk, the growing size and complexity of biological image data sets precludes manual analysis and calls for increasingly advanced automatic algorithms that are generic enough to be easily tunable to different applications, yet robust enough to deal with different cell type and strongly varying imaging

13 5 Figure 2.2. Hypothetical lineage tree of one E. coli cell. The goal is to construct a lineage tree for all cells in the video. Source: Q. WANG, J. NIEMI, C. TAN, L. YOU, AND M. WEST, Image segmentation and dynamic lineage analysis in single-cell flourescence microscopy, Cytometry A, 77A (2009), pp conditions. [6, p. 185] An accurate automated method would improve data processing capability and remove operator error present in manual segmentation of large cell populations. The key to an automated method is to recognize individual cells, regardless of their shape, size, and appearance relative to the varying background, track their growth over successive frames, and construct a lineage tree relating cells from frame to frame. While there are numerous methods proposed for general object tracking, cellular videos pose a challenge to those existing techniques due to severe image noise and clutter, cell shape deformation, and image contrast change [23, p. 390] which are often not present in static imagery. Tracking methods are most accurate when used to identify objects in an environment where the background and object target appearance remain constant. Time lapse microscopy video introduces inconsistent background through successive frames. As such, the use of edge-based cell detection and tracking methods can be unreliable. In addition, the close proximity of cells, possible cell occlusion, unpredictable cell trajectory, and narrow focal plane increase the complexity of the problem. Many existing methods deal with cell division and movement well, but the presence of varying backgrounds which introduce unpredictable noise often cause false detection and tracking to occur. A universal automated algorithm for cell segmentation and tracking has yet to be developed.

14 6 2.3 IMAGE SEGMENTATION The success of any generational tracking method relies on the accuracy of the segmentation algorithm. Individual cells must be detected in each frame before tracking and frame to frame object correlation can be conducted. When cells are tightly clustered, which is often the case after multiple rounds of E. coli cell division, it is typically hard to uniquely identify individual cell objects. While many methods ignore the boundary detection task and track cells via centroid relocation, using intensity detection techniques and neighborhood based object correlation [23], the most effective methods employ an image segmentation step. This paper focuses on the image segmentation (i.e., individual cell detection) task of the generational tracking problem. Many image segmentation techniques have been employed for the detection of objects in imagery. These methods include Otsu s threshold selection method [12] utilizing cumulative moments of the gray-level histogram to separate objects from surrounding background without a priori knowledge of the objects or image [12]. However, the distinction between object and background may not be consistent throughout the image. Selection of a universal and accurate threshold for segmentation is difficult in these situations. The watershed transform [19] is another image segmentation tool that has been applied to address the cell detection and tracking problem. This method utilizes the gradient image as a topological surface to identify gradient maxima and minima which correspond to object boundaries. However, this method often leads to an over-segmentation of the image with many cells segmented into multiple regions. Finally, edge detection routines such as the Canny edge detector [16] have been employed to automatically detect the edges of objects in an image. However, like many gradient dependent methods, the boundaries between objects and background may not be distinct enough to support accurate segmentation using such methods. The weak boundaries of E. coli cells cannot be extracted by popular edge detectors [23]. Active contour based algorithms have proven to be very suitable for detection of different cell types with varying image conditions. However, traditional active contour methods can be highly inaccurate when segmenting cluttered images and are computationally expensive. The level set method addresses many of the topology changes that other segmentation methods cannot handle. In this paper, we explore various versions of an active contour, level set, method for image segmentation. Many argue that the close proximity of cells and the occlusion in microscopy videos makes edge-based cell tracking difficult. [23] To address these concerns, we explore a non-gradient based level set implementation which is successful in detecting objects in low-quality, congested images. We present an advanced multiphase level set method for image segmentation with new approaches to algorithm termination and identification of individual cells.

15 7 CHAPTER 3 METHOD BACKGROUND Image segmentation can be formulated as a problem of modeling the motion of a propagating front according to a curvature dependent speed. If moved at the appropriate speed, the front will stop at all object boundaries in the image space. Traditional numerical methods for modeling these surfaces have many weaknesses. They often require parametrization of the moving front and discretization of the front boundary to a finite set of points. The discrete points need to be updated with each iteration of the method, which adds steps and computation time. While these methods are extremely accurate for modeling simple motion, they fail when dealing with more complex motion problems. For the most part, these methods are unable to efficiently and accurately handle changing topology, such as the splitting of a region, overlap, or the introduction of corners. To handle such changes, traditional methods will re-grid the surface, reinitialize the discrete points, and often introduce new points. The re-gridding technique introduces error that compounds over time and the reinitialization takes time and introduces additional computations to the problem. Other methods model the behavior of the front by discretizing and following the interior of the surfaces. These methods handle topological changes better, but face difficulty when trying to calculate the curvature of the front. 3.1 GENERAL LEVEL SET METHOD The level set method follows the propagating front approach to image segmentation. This method simplifies computations involving complex surfaces and curves, including those with time-varying topology, by following the motion of an (n 1)-dimensional surface using an n-dimensional underlying function φ. Unlike most numerical approaches, the level set method is implemented in an Eulerian Framework (fixed coordinate system). As such, the number of computational elements is fixed and will not change throughout the iterations. More importantly, the method enables numerical computations involving complex surfaces on a fixed grid without parametrization [11]. The level set method can be utilized in image science for segmentation, repair, inpainting, and object detection. In partial differential equation (PDE) based image processing methods, images are characterized as a continuous function sampled on a grid (pixels) [13]. The original image is labeled u 0. In general form, the PDE method for image applications is

16 8 u t + Lu = λru with L an operator and R a regularization term [13]. All such methods are centered around the processing of the level lines (or level sets) of the image. The level set method uses a set of PDEs to model the movement of a surface and finite difference methods to numerically approximate solutions to the set of PDEs. The scheme is implemented to follow the motion of an (n 1)-dimensional surface using an n-dimensional underlying function φ (Figure 3.1) [15]. Figure 3.1. Level set method illustration. Red region represents level set φ, blue represents xy plane, gray areas are bounded regions. The boundary of the regions is the zero level set of φ. All points on the xy plane that are inside the the boundary are the points where φ > 0 and the points outside the boundary are those where φ < 0. Source: UNK, Level set method. Wikipedia, set method, accessed August 2011, August The level set method is used to separate the image domain into different regions. The boundary of the regions in the image are defined as the zero level set of the underlying function φ. More specifically, the xy plane is placed at the zero-level of φ. All points on the xy plane that are inside the boundary correspond to the points where φ > 0 and the points outside the boundary are those where φ < 0. Figure 3.1 illustrates how this method can easily deal with changing topology. In traditional numerical methods, the splitting or merging of a region would require the algorithm to identify the change, re-parameterize the boundary, introduce additional boundary points, and increase the complexity of the algorithm. The original level set method was developed based on numerical analysis of known equations of motion and the behavior of curves and moving surfaces. Osher and Sethian

17 9 determined an initial-value Hamilton-Jacobi like equation, first order, non-linear partial differential equation in the form of φ t + H(x, Dφ) = 0, to model the movement of an n-dimensional surface [11]. For the level set application, the H(x, Dφ) depends on the curvature and speed of the propagating front. The level set method follows surface changes by modifying the location of the zero-level set on the underlying function φ. A general level set algorithm consists of three main steps [13]: 1. Initialize level set φ at t = 0 2. Construct and approximate H(t, x, φ, Dφ, D 2 φ...) = 0 3. Evolve the level set φ t + H(t, x, φ, Dφ, D 2 φ...) = 0 for t + 1 The evolution, step 3, is implemented by a numerical approximation method, such as finite differencing. The original model, developed by Osher and Sethian, serves as the basis for most level set applications [11]: φ t F (κ) φ = 0, with initial data φ(x 1, x 2,...x n, t = 0) = φ(x, 0) = φ 0 (x) = Γ(0), where Γ(0), called the zero level set of φ, is an arbitrary initial contour in the fixed grid region and F (κ) the curvature dependent velocity [11]. The most important feature of this method is that the equation for φ in n space variables can be applied regardless of whether the propagating front of interest can be explicitly written as a function. The initial value and boundary values are based on the chosen initial level set. The method can be applied to higher dimensions, but for our study we focus on two dimensions: φ t (x, y) F φ(x, y) = 0, Γ = {(x, y) φ(x, y) = 0} Γ(0) = φ(x, y, 0) = φ 0 (x, y). The arbitrary initial contour is propagated along its gradient according to the three step process above. If the contour is moved at an appropriately selected speed F, it will stop (after a finite number of iterations) at the boundary of all objects. 3.2 WEAKNESSES OF GENERAL METHOD The original level set methods display some weaknesses. These traditional methods depend on the gradient of the image and propagating contour, which introduces error and inaccurate segmentation in many applications. Specifically, objects not defined by the gradient will not be detected by these methods. In addition, the traditional schemes define the

18 10 initial level set function as a signed distance function [13]. Furthermore, it is critical that the evolving function remain close to the signed distance function throughout the iterative process. However, the basic gradient based level set methods will often deviate from the signed distance function during the evolution process and develop steep or flat gradients, which will cause problems for numerical approximation [4]. In an effort to avoid these problems, a distance reinitialization step is implemented. The level set function is reshaped while the zero level set location is maintained. The reinitialization evolves the curve to steady state where φ becomes the signed distance function to the zero level set. While the reinitialization process is normally only needed for a small neighborhood around the zero level set, the additional steps complicate and slow down the overall method [10]. More advanced level set implementation methods, avoid the need for periodic re-initialization [9]. Some methods force the level set function to remain close to a signed distance. These methods include a term in the energy functional which penalizes deviation from the signed distance function [10]. Other methods, such as our multiphase level set method which is introduced in Chapter 4, do not require a relative signed distance function contour [3]. These methods are designed to converge to the energy functional global minimizer independent of the choice of initial contour. 3.3 MUMFORD-SHAH SEGMENTATION The image segmentation problem can be approached in many ways. Mumford and Shah [20] formalized the segmentation problem as the minimization of an energy functional (3.1). For this problem, we let Ω R n be open and bounded and define C as a closed subset in Ω made up of a finite set of smooth curves. The connected components of Ω \ C are denoted by Ω i, such that Ω = i Ω i C. The function u 0 : Ω R is the given bounded image-function. The Mumford-Shah segmentation problem is defined as given an observed image u 0, find a decomposition Ω i (connected components) of Ω (open and bounded) and an optimal piecewise smooth approximation u of u 0, such that u varies smoothly within each Ω i, and rapidly or discontinuously across the boundaries of Ω i [20]. The minimizer of the energy functional would identify the approximate edges of u 0. The solution, u, is formed by regions Ω i with boundary C. ( F MS (u, C) = α u 2 + β u u 0 2) dxdy + length(c). (3.1) Ω\C The problem can be further simplified by restricting the segmented image to piecewise-constant functions inside each component Ω i [3]. In this reduced form, the problem becomes a minimal partition problem. A new energy functional (3.2) is minimized to

19 11 determine the boundary of the object region. F MS (u, C) = u 0 c i 2 dxdy + ν C. (3.2) i Ω i It turns out that this energy functional is minimized in c i by setting c i = mean(u 0 ) in Ω i [4]. Therefore, the minimization is with respect to the set of region boundaries. It is not easy to minimize energy functionals that represent regions and their boundaries in practice. The energy functional (3.2) can have many local minima [1]. Therefore, inaccurate minimization routines will often stop at a local minimum instead of the global minima. To improve the performance of the minimization method, both local and global information must be incorporated. Active contour methods, specifically level set based segmentation methods, are local segmentation approaches that take region information into consideration. These methods incorporate information from both inside and outside the evolving contour. Movement of the contour is regulated by the changing characteristics of the distinct image areas. Specifically, the active contour model is a variational framework (i.e., minimizes an energy functional) for extracting object boundaries in an image by evolving a contour subject to constraints of the image. 3.4 VARIATIONAL LEVEL SET METHOD The level set method can be constructed as a variational active contour problem, in which the solution to the system of PDEs governing the level set evolution minimizes an energy functional. The energy functional commonly used is similar to the Mumford-Shah functional. The global minimum of the energy functional, traditionally determined by gradient-descent minimization, corresponds with the object boundary. The evolving contour is defined as a Lipschitz continuous function φ such that the boundary of the internal (object) region and external (background) region, at the energy functional minimum, corresponds to φ(x, y) = 0. The variational level set methods produce robust results because they allow additional region information to be incorporated into energy functionals. For example, Li et al [10] developed a variational level set method that minimizes an energy functional E(φ) = µr p (φ) + E m (φ), which depends on image data as well as the active contour φ. The functional has an internal energy term R p (φ) = p( φ )dxdy, Ω

20 that penalizes the deviation of the level set from a signed distance function and an external energy term E m (φ) = E g,λ,ν (φ) = λl g (φ) + νa g (φ), L g (φ) = gδ(φ) φ dxdy and A g (φ) = gh( φ)dxdy, Ω that drives the motion of the zero level set toward the image objects [9]. The functions H(φ) and δ(φ) represent the common Heaviside and Dirac delta length and area approximation functions. The method keeps the evolving contour close to a signed distance function and does not require re-initialization. It is an improvement over traditional methods as it implements a larger time step, which speeds up evolution, can be initialized with a general function rather than the signed distance function, and the numerical approximation for the evolution step requires a much simpler finite difference scheme (see Appendix A and B.1). As such, their approach was the first method we implemented to address the E. coli cell segmentation problem. Despite the promising results presented by Li et al [10], their variational method performed poorly on the E. coli image frames. The method was unable to identify the unique cell boundaries and extracted the cell mass as one object. In addition, the method required a significant amount of iterations to converge to the boundary of the cell mass. Once the boundary of the cell mass was reached, additional iterations forced the level set to pass the edge of the cell mass. No number of iterations resulted in identification of the individual cell boundaries. As such, this variational level set method proposed by Li et al was abandoned for our application. Details of this failed approach are included in Appendix A, with illustrated results in Figure A.1. The main weakness of this method was its dependence on the image gradient. Like most level set methods, the failed method depends on an edge indicator function, g = G σ I 2, with G σ the Gaussian kernel with standard deviation σ, to stop the evolving curve on the object boundaries. These type of models are only able to detect objects with edges defined by the gradient [4]. Objects in an image with edges not defined by the gradient will not be detected. As a result of the observations made with failed methods (see Appendix A, Figure A.1), we conclude a gradient-based method will not be accurate for our application. The variational level set approach is not completely abandoned for this application. Chan and Vese proposed an active contour model using a variational level set formulation with a stopping term based on the Mumford-Shah functional (Equation 3.1), as opposed to the image gradient [3]. The model assumes the image u 0 is formed by two regions of Ω 12

21 13 piecewise-constant intensities, u 1 0 and u 2 0, one representing the object region and one representing the background. The boundary of the object region, C, is the minimizer of an energy functional (Equation 3.3) which combines information from inside and outside of C as well as regularizing terms. F CV (φ, c 1, c 2 ) = µ δ(φ) u + ν H(φ)dxdy + λ 1 u 0 c 1 2 H(φ)dxdy Ω Ω Ω + λ 2 u 0 c 2 2 (1 H(φ))dxdy. (3.3) Ω The energy functional governs the evolution of the active contour and will be minimized when the evolving contour reaches the boundary of the object, i.e., φ = C. Chan and Vese derived the specific energy functional (Equation 3.3) as a case of the Mumford-Shah minimal partition problem for image segmentation (Equation 3.1). This model can handle more general types of boundaries and will be the basis of our detailed level set application.

22 14 CHAPTER 4 METHOD Methods with a single level set function φ can only model two distinct regions (called phases) in an image. The initial variational level set method implemented for this application, as developed by Li et al, illustrates this limitation. In addition, single level set functions cannot represent some geometrical features of the boundary, such as overlap or triple junctions. For more complicated situations, the two phase active contour model is insufficient and additional level set functions need to be introduced to detect all regions in the image [2]. 4.1 MULTIPHASE LEVEL SET The active contour model for solving the minimal partition problem (Equation 3.3) can be extended to partition an image into more than two distinct regions of piecewise-constant intensities. By incorporating additional level set functions, a multiphase level set method can detect additional phases in the image. Some multiphase methods use a separate level set function φ i to represent each phase Ω i in the image domain Ω. However, these methods are computationally intensive and cannot handle overlap, vacuum, or triple junctions. The Chan-Vese multiphase level set model, which is the basis of our application, can represent up to n phases with complex topologies using only m = log 2 n level set functions. The Chan-Vese multiphase level set method can identify individual segments in images with multiple segments and junctions, as compared with the initial model (two phase Chan-Vese model), where the detected objects were belonging to the same segment. [20, p. 275] In general, we define the boundary of an open set ω Ω as C = ω. In the level set method we represent this curve as the zero-level set of a Lipschitz continuous level set function φ : Ω R. φ(x, y) > 0 in ω φ(x, y) = 0 on ω φ(x, y) < 0 in Ω \ ω The curve length C and the area of the phase region ω can be approximated using functionals depending on the well-known Heaviside and Dirac delta functions. These approximations appear in all level set methods. In practice, the general Heaviside and Dirac delta functions are regularized (Equations 4.3 and 4.4) by a H ε that converges as ε 0 [2]. The approximation functionals (Equations 4.1 and 4.2) are minimized as part of the level set

23 evolution process. C L ε (φ) = Ω H ε (φ) dxdy = ω A ε (φ) = H ε (x) = 1 2 Ω Ω 15 δ ε (φ) φ dxdy (4.1) H ε (φ)dxdy (4.2) [ π arctan ( x ε ) ] (4.3) δ ε (x) = H ε(x) = 1 ε (4.4) π ε 2 + x 2 The generalization of the two phase active contour model, the multiphase model, considers m = log 2 n level set functions φ i : Ω R to represent n phases. The union of the zero-level sets of φ i, the set of curves C, will represent the edges in the segmented image [20]. In this method, the distinct phases are disjoint and their union is the domain of Ω. Specifically, a phase is defined as the set of pixels (x, y) of Ω with the same vector Heaviside function (4.5) value; each pixel will belong to only one phase in Ω. Φ = (φ 1, φ 2,..., φ m ) H(Φ) = (H(φ 1 ), H(φ 2 ),..., H(φ m )) (4.5) Since there are n = 2 m possibilities for vector 4.5, there can be up to n = 2 m distinct phases in Ω [20]. Therefore, as mentioned above, only m = log 2 n level set functions are required to represent n phases. We label the distinct phases by I with 1 I 2 m = n. Based on the piecewise-constant two phase Mumford-Shah minimization problem (3.2), we know the generalized multiphase energy functional will be minimized by a set of constants c i corresponding to the mean of u 0 in the region i. Accordingly, we introduce a constant vector of averages c = (c 1, c 2,..., c n ) where c I = mean(u 0 ) in the class I. The generalized multiphase energy functional to be minimized is F n (c, Φ) = (u 0 (x, y) c I ) 2 χ I dxdy 1 I n=2 m + 1 i m ν Ω Ω H(φ i ), (4.6) with χ I the characteristic function for class I. This characteristic function incorporates information regarding the approximate curve length and area inside and outside of each phase I. This general method can be expanded to additional dimensions, for processing RGB image data, but for our application we focus on the two dimensional application.

24 FOUR PHASE LEVEL SET METHOD The initial variational level set application to E. coli cell division video frames (see Appendix A) indicated that the two phase level set model was inadequate for individual cell detection. The presence of dark cell regions, bright pixels between cells, and background caused the two phase method to detect the cell mass as one region of constant intensity. There were more distinct regions than the two phase model was able to identify, so some regions were lumped together as one. As such, the E. coli video frame segmentation problem requires additional level set functions for accurate cell detection. The Four Color Theorem [18] states that given any separation of plane into contiguous regions, no more than four colors are required to color the separated regions so that no two adjacent regions have the same color [18]. Based on this theorem and observations from the two phase level set method, we determine that four colors (or phases) are required to separate the distinct cells from the background and surrounding white noise. Accordingly, the focus of our specific application is the four phase piecewise-constant level set method, partitioning the image into four regions (Figure 4.1) [20]. Using two level set functions, we define four different regions by disjoint sets: {φ 1 > 0, φ 2 > 0}, {φ 1 > 0, φ 2 < 0}, {φ 1 < 0, φ 2 > 0}, {φ 1 < 0, φ 2 < 0}. The boundaries of the regions are defined by the two curves {φ 1 = 0} and {φ 2 = 0}. The union of these curves is the boundary C. Figure 4.1. Partition of the image domain into four regions using two level set functions. Two curves: {φ 1 = 0} and {φ 2 = 0}. Four regions: {φ 1 > 0, φ 2 > 0}, {φ 1 > 0, φ 2 < 0}, {φ 1 < 0, φ 2 > 0}, {φ 1 < 0, φ 2 < 0}. Source: L. A. VESE AND T. F. CHAN, A multiphase level set framework for image segmentation using the Mumford and Shah model, Int. J. of Comp. Vis., 50 (2002), pp

25 4.6, is: For m = 2 level set functions (n = 4 phases), the energy functional, which is based on F 4 (c, Φ) = (u 0 c 11 ) 2 H(φ 1 )H(φ 2 )dxdy Ω + (u 0 c 10 ) 2 H(φ 1 )(1 H(φ 2 ))dxdy Ω + (u 0 c 01 ) 2 (1 H(φ 1 ))H(φ 2 )dxdy (4.7) Ω + (u 0 c 00 ) 2 (1 H(φ 1 ))(1 H(φ 2 ))dxdy Ω + ν H(φ 1 ) + ν H(φ 2 ) Ω The energy functional being minimized with this method (Equation 4.7) is non-convex and may have local minimizers [1]. The algorithm could falsely converge to these minimizers, under certain conditions. As such, the parameters for evolution must be chosen appropriately to ensure that the algorithm converges to the global minimizer. The constants c = (c 11, c 10, c 01, c 00 ) obtained by minimizing Equation 4.7 are defined by the image mean in each of the four phases (Equations ). Ω c 11 (Φ) = mean(u 0 ) in {(x, y) : φ 1 (t, x, y) > 0, φ 2 (t, x, y) > 0} (4.8) c 10 (Φ) = mean(u 0 ) in {(x, y) : φ 1 (t, x, y) > 0, φ 2 (t, x, y) < 0} (4.9) c 01 (Φ) = mean(u 0 ) in {(x, y) : φ 1 (t, x, y) < 0, φ 2 (t, x, y) > 0} (4.10) c 00 (Φ) = mean(u 0 ) in {(x, y) : φ 1 (t, x, y) < 0, φ 2 (t, x, y) < 0} (4.11) The Euler-Lagrange equations (4.12 and 4.13), which evolve φ 1 and φ 2, are obtained by minimizing 4.8. dφ 1 dt ( ) φ1 = δ ε (φ 1 ){ν div φ 1 [((u 0 c 11 ) 2 (u 0 c 01 ) 2 )H(φ 2 ) + ((u 0 c 10 ) 2 (u 0 c 00 ) 2 )(1 H(φ 2 ))]} (4.12) 17 dφ 2 dt ( ) φ2 = δ ε (φ 2 ){ν div φ 2 [((u 0 c 11 ) 2 (u 0 c 01 ) 2 )H(φ 1 ) + ((u 0 c 10 ) 2 (u 0 c 00 ) 2 )(1 H(φ 1 ))]} (4.13) The solutions to this system of differential equations correspond to the minimization of 4.8. The curves φ 1 and φ 2 are evolved toward the boundaries of the four phases based on

26 the solutions to this system. A numerical algorithm with semi-implicit finite differencing (detailed in Appendix B.3) and no-flux boundary conditions is used. At each iterative step t, the values c t 00, c t 01, c t 10, and c t 11 are re-computed and then the curves are evolved according to φ t+1 1,i,j and φt+1 2,i,j. Evolution stops when the curves are stationary at the boundaries of the regions and the global minimum of the energy functional is identified INITIALIZATION For the single level set curve implementation, the initial contour φ 0 is defined as any simple contour in the two dimensional image space Ω. The initial contour can be defined as a single point, a small contour on the outer edge of the region, a large contour surrounding the region, or a collection of simple contours (ex: multiple distinct circles in the domain). Typically, the initial function is chosen to resemble a binary step function (4.14). c 0 x Ω 0 Ω 0 φ 0 ( x) = 0 x Ω 0 (4.14) c 0 x Ω Ω 0 The constant c 0 > 0 is arbitrary. The initial level set function is the binary step function for an arbitrary region Ω 0 in the image domain Ω. The points inside the contour region (Ω 0 Ω 0 ) are initialized positive, the points on the boundary of the region ( Ω 0 ) are initialized zero, and the points outside the arbitrary region (Ω Ω 0 ) are initialized negative. The multiphase level set method is not dependent on the choice of the initial level set function. However, there are three commonly used types of initialization, which resemble Equation 4.14), that are known to perform well on a variety of images (Figure 4.2) [20]. Figure 4.2. Level set initialization types. (a) Type 1 (b) Type 2 (c) Type 3. Source: L. A. VESE AND T. F. CHAN, A multiphase level set framework for image segmentation using the Mumford and Shah model, Int. J. of Comp. Vis., 50 (2002), pp

27 19 The contours φ 1 and φ 2 are initialized in a manner similar to (4.14). Specifically, regardless of the type of initialization selected, we set φ 0 1 and φ 0 2 according to the equations in c 0 x Ω 1 Ω 1 c 0 x Ω 2 Ω 2 φ 0 0 1( x) = 0 x Ω 1 φ2( x) = 0 x Ω 2 (4.15) c 0 x Ω Ω 1 c 0 x Ω Ω 2 As illustrated in Figure 4.2, we choose from one of three initialization types for our implementation: Type 1: Two large, overlapping initial contours. The overlap will allow the φ 1 (red) spread into regions enclosed by φ 2 (blue) in fewer iterations. However, the overlap can cause the process to converge to a local minima rather than a global minima on some images. This will result in some areas not being enclosed by either level set function if the evolution process is stopped too soon. Type 2: Two large, non-overlapping initial contours. The non-overlapping curves will often miss object regions if the evolution process is stopped too soon, especially if a region is not in contact with the inside of either level set at initialization. Type 3: Two small, overlapping regions repeated throughout the image space. This initialization performs the best, especially on noisy or congested images. The numerous small contours allows the method to converge to an accurate minima value in the fewest iterations. We find that type 1 initialization often requires a large amount of iterations and is highly inefficient. As such, we do not consider this initialization type in our testing. We include this in our discussion since it is considered in many of the referenced papers. 4.4 STOPPING CRITERION The evolving level set functions will navigate around topological structures in the image space and move toward the boundaries of objects. When the boundaries are reached the contours should stop evolving [4]. Iterative loops are often automatically terminated when a maximum number of iterations is reached, as defined by the user. We find this minimal stopping criterion to be highly inefficient and inaccurate for the level set method. Results of initial testing indicated that the contours will often evolve past the true boundaries of the objects before the maximum number of iterations is reached, causing erroneous results. An accurate and efficient choice for the maximum number of iterations requires some a priori knowledge about the performance of the method on each image. In

28 20 order to produce accurate results without a priori knowledge, the proper iteration number (less than the maximum iteration threshold) must be identified with an initial run. Each individual image will likely require different iterations for convergence to the true global minima energy functional. To improve the traditional stopping criterion, we implement a curve length criterion [22]. When the evolving curves reach the true boundary they should stop or slow down for a number of consecutive iterations. A simple evolving curve length criterion [21] will automatically stop the evolution process if the combined level set curve length L(C) = L(z 1 ) + L(z 2 ), with z 1 = {(x, y) φ 1 (x, y) = 0} z 2 = {(x, y) φ 2 (x, y) = 0}, has changed less than a length threshold λ L for a specified number of consecutive iterations, λ T. 4.5 CONNECTED COMPONENTS LABELING The construction of a generational lineage tree requires the ability to automatically distinguish individual cells. The multiphase level set method will separate regions of like features, up to four different regions with a two phase level set method. The general multiphase level set method will separate E. coli cells from the background and other objects in the image, but it will not separate individual E. coli cells with the same image characteristics. In order to separate homogeneous populations of E. coli cells, we implement connected components labeling [7]. When the evolution process has stopped, the individual cells will be identified and counted. First, a binary image separates the cell pixels from the background pixels. This binary image is then scanned pixel by pixel (left to right, top to bottom). The background pixels are assigned to image region 1. We move along each a row until a non-background pixel (p) is reached. We examine the four neighbors of p which have already been processed [7]. These include the pixel to the left, above, and the upper left and right diagonal pixels (Figure 4.3a). If all four neighbors are background pixels, we assign a new cell region label to p and increase the cell region count by one. If only one neighbor belongs to a cell region, then we assign p to that cell region. Otherwise, if more than one neighboring pixel is part of a cell region, we assign p to the closest cell region using the neighbor pixel rank (Figure 4.3a). A final labeling evaluation is conducted after all pixels have been assigned to a region (background or cell). If any region contains fewer than ten pixels (arbitrary small cluster of

29 21 Figure 4.3. Connected components labeling method. (a) Neighboring cell rank. (b) Secondary neighboring cell rank. pixels), we expand our search around these pixels to include additional neighbors (Figure 4.3b). The pixels within a small cluster will be individually assigned to any large cell region within the expanded neighborhood. If no neighboring cell region is identified, the small cluster is assumed to be image noise and is assigned to the background region (i.e. filtered out). This cluster threshold is determined relative to the image resolution and size as to avoid filtering out cells in smaller images. When all the pixels have been assigned to a region, background or individual cell, the labeling function will count the number of individual cells in the image region. We show, in the results section, that this technique is able to accurately connect and label cells undergoing cell division, with the septum pinching in, as well as non-dividing cells. This labeling capability will enable automated cell comparison between image frames. Using the output of the labeling function between successive frames, k, k + 1, k + 2, etc, associations between cells can be constructed to form a basic frame to frame lineage tree. A detailed method flow chart, illustrating the processing steps, is provided in Appendix C.

30 22 CHAPTER 5 RESULTS Bacterial cell division occurs rapidly and results in an exponential growth of cells between frames of a microscopy video. The cell population mass often gathers around the initial cells and grows in the outward direction. Therefore, a cell cluster develops at the center of each video frame, which results in little separation between distinct cells. Traditional level set methods fail to accurately extract the individual object boundaries in congested video frames. Often the large mass of cells is detected as one object or the white noise between cells, which contrasts most from the background, is detected. The results that follow show that the boundaries of individual cells can be successfully extracted by the multiphase level set method, despite the presence of congestion and white noise. The proposed piecewise-constant multiphase level set method is been applied to a variety of synthetic test data and real microscopy video frames. The experimental results show that the method performed well on both the ground truth test data, from the work of Chan and Vese [20], as well as on our E. coli cell division video data. Only minor modifications to parameters were required to maintain performance levels between distinct images or video frames. For each implementation in this chapter, we fix the space steps h = x = y = 1 and the time step t = 0.1 with ε = h (for δ ε and H ε ). The time step t determines the size of the evolution time step as the contours φ 1 and φ 2 are evolved within each frame for cell detection. This time step is not related to the time difference between successive frames of the video. It was determined that as the bacterial growth frames became more congested with E. coli cells, the initial level sets, the value of ν, and the stopping criterion had to be modified to produce accurate results. The latter two facts were supported by the published results of Chan and Vese [20], in which the more complicated images required additional level set evolutions as well as a more sensitive ν. We find that ν = 0.01 x works well for the less congested E. coli frames while ν = x works well for the video frames with more E. coli cells present. Similarly, a value of λ L = 5 is sufficient for early frames whereas a value of λ L = 2 is required for the later, more congested frames. These parameter values can be dynamically initialized relative to the frame number k and number of cells in frame k 1. We focus on the four phase model with two level set functions. We implement a slightly modified Chan-Vese piecewise constant multiphase level set method. Similar to Chan-Vese, we initialize the level set functions φ 0 1 and φ 0 2, compute the region averages (c I ),

31 23 and solve one step of the PDEs in φ 1 and φ 2. The latter two steps complete the contour evolution step and are iterated in a loop until the boundaries of the objects are extracted. When the evolution process has stopped, our method will identify and count the distinct cell regions extracted by the level sets using connected components labeling. 5.1 GROUND TRUTH RESULTS: TEST DATA PROCESSING In an effort to validate our implementation of the piecewise constant multiphase level set method, as developed by Chan and Vese, we show numerical results using our multiphase model on data used in their paper [20]. We use the modified stopping criterion in addition to the basic maximum iteration criterion for our implementation of the method. Applying our method to known ground truth data allowed us to test the performance of our model as well as observe the influence of the initial conditions on the behavior of the model. We show that our modified model reproduced similar results. In addition, we show that it works to efficiently identify objects in the image space using minimal iterations. Once our method replicated the final level set evolution results for the test data, the labeling functions were applied to identify and count the total number of distinct objects in the image space. The performance of these functions on the simple test data indicated appropriate initial conditions for the more complicated bacteria growth video frames. The two phase level set method is able to identify up to four distinct image regions (i.e, regions of the image with similar characteristics and appearance). We begin with two synthetic images containing four regions (background and three objects). We consider type 2 initialization for the first image and type 3 initialization for the second image with a two phase piecewise-constant model. For both implementations, we use ν = x and λ L = λ T = 5. We allow a maximum of 600 iterations. Figure 5.1 shows the evolution of the contours on the first test image. As we can see, all but a small portion of one of the objects is extracted by the evolving contours. The contours are stopped by the stopping criterion at 185 iterations, which indicates that the length termination criterion was met. The only improvement to this result, as shown by Chan and Vese, is made by superimposing the initial contours, which will then behave as one active level set function. The final level sets separate the regions identified in the image. The background is identified as the pixels where φ 1 < 0 and φ 2 < 0. The cell pixels are identified as any pixel where φ 1 > 0 or φ 2 > 0. There will often be an overlap of cells detected by φ 1 and those detected by φ 2. These overlapping pixels are contained in the region where φ 1 > 0 and φ 2 > 0.

32 24 Figure 5.1. Evolving contours overlaid on original test image 1. Type 2 initialization with h = x = y = 1, ε = h, t = 0.1, ν = 0.01 x 255 2, λ L = λ T = 5. (a) Initial level set contours. (b) Level set contours after 25 interations. (c) Final level set contours after 185 iterations. The connected components labeling and counting function is applied to the final level sets to identify and count the distinct non-background regions in the original image. We see the method accurately separates the background from the object regions (Figure 5.2a) and determines there are three distinct non-background regions in the original image (Figure 5.2b). Figure 5.2. Test image 1 results. Final contour evolution and connected components labeling for test image 1. (a) Binary background v. object pixel output. (b) 3 distinct object regions detected. The small contours making up the level set functions in type 3 initialization often allow the method to converge to a global minimum more rapidly than other initialization types. Figure 5.3 shows the evolution of the contours on the second test image using type 3 initialization. We see that the termination criterion is met, stopping the evolution process, after only 50 iterations of the curves. Similar to the previous results, our connected components labeling and counting function accurately identify three distinct non-background regions in the original test image 2 (Figure 5.4).

33 25 Figure 5.3. Evolving contours overlaid on original test image 2. Type 3 initialization with h = x = y = 1, ε = h, t = 0.1, ν = 0.01 x 255 2, λ L = λ T = 5. (a) Initial level set contours. (b) Level set contours after 25 interations. (c) Final level set contours after 50 iterations. Figure 5.4. Test image 2 results. Final contour evolution and connected components labeling for test image 2. (a) Binary background v. object pixel output. (b) 3 distinct object regions detected. 5.2 CONCLUSIONS OF INITIAL TESTING The results of the initial testing, with known ground truth data, validated the performance and accuracy of our multiphase level set method with efficient stopping criterion. The trials on synthetic data during development identified appropriate choices for ν, λ L and λ T to ensure accuracy and speed. Specifically, we determined that λ L <= 5 and λ T <= 10 are acceptable choices, with λ L = λ T = 5 performing well for most images. During trials, we observed the influence of ν on the evolving curves. This parameter is a coefficient of the length term. A smaller value of ν results in smaller changes in the level set evolution at each step. A larger value of ν forces the level sets to make larger, more extreme movements with each iteration. The smaller ν requires additional iterations for convergence, but it is more sensitive to image features. As such, we concluded noisier and more congested

34 26 images will require a smaller ν to prevent the level sets from passing through small region boundaries. The initial testing showed that our implementation can detect interior contours, complex topologies, and sharp edges without reinitialization. When implemented, our counting functions automatically determine the number of distinct, connected objects in the region. These functions would only be applied to images that contain easily distinguishable and countable regions. The numerous published results of level set applications indicate that the method is not directly dependent on the choice of initial contours. Any chosen initial contour will eventually extract the object boundaries under the right parameters. However, we found that the choice of the initial level set functions is critical to the efficiency of the method. Based on observations from above, we determined that either type 2 or type 3 initialization would ensure rapid identification of all cells in the congested E. coli video frames. More generally, we determined that at least one of the initial level set contours must contact or surround each of the objects in the region in order for them to be identified without the need for excessive iterations. However, it should be noted that most any initial level set functions will eventually identify the objects in the region, given enough iterations. 5.3 BACTERIAL GROWTH RESULTS The modified method, with information gained from the ground truth testing, is applied to individual frames of a time lapse phase-contrast E. coli cell division video [14]. The E. coli cells appear as dark regions surrounded by a white halo in this phase-contrast video format. The cell mass clusters at the center of the focal plane and grows in an outward direction as cell division occurs. The data was obtained from a public website and is used as a proof of concept for this application. The video frame rate and pixel resolution is unknown. Each 240 x 320 frame is processed individually. The results of the level set evolution and connected component labeling functions are displayed in figures that follow. The results of each frame were stored and compared with successive frames to construct a simple frame to frame lineage tree. We begin with a few basic frames containing few cells before considering more complicated, congested frames. Based on knowledge gained in initial testing, we consider both type 2 and type 3 initialization. We find that as the number of cells increases (i.e., the frame number increases), the initial conditions and function parameters need to be modified accordingly. Frames with an abundance of cells required a smaller ν as well as a smaller λ L. If we used a constant ν for all frames, the level sets moved too quickly and missed cells in the congested image frames. The presence of many cells required numerous small adjustments to the contours as they

35 27 slowly moved around the individual cell regions. As such, we restricted the length criterion to λ L < 5 for later frames to prevent the evolution process from stopping too soon. The evolution iterations will not be stopped until the overall contour length change is significantly small relative to the assumed frame cell population. These parameters could be dynamically defined based on the current frame number k and relative to the number of individual cells identified in frame k SIMPLE DETECTION: SMALL POPULATION OF CELLS We apply our modified multiphase level set method to frames 1 and 6 of the E. coli cell division video (Figure 5.5) [14]. We use type 3 initialization, for fast convergence and detection of small regions. We use ν = 0.01 x 255 2, to capture the fine detail of the cells, and we let λ L = λ T = 5. Figure 5.5. Initial video frames. These contain few E. coli cells for simple detection. (a) Frame 1. (b) Frame 6. Source: UNK, Bacteria growth video. accessed February 2012, July Our method correctly extracts the cells in the initial frames under these parameters. Figure 5.6 shows the segmentation process of the first frame of the video. Despite the close proximity of the cells to each other and the presence of white noise, the cell regions are extracted from the background. The evolution is stopped at the true boundaries of the cells after only 50 iterations (Figure 5.6c). The objective of our connected components labeling function is to uniquely identify the cells in the cell region. We easily separate the background from the cell region using the results of φ 50 1 and φ 50 2 as shown in Figures 5.7a and 5.7b. In Figure 5.7a we see the background (original image) vs. the cell region (white). In Figure 5.7b, the blue pixels represent background and the red pixels represent the cell region. These results are provided

36 28 Figure 5.6. Evolving contours overlaid on original frame 1. Type 3 initialization with h = x = y = 1, ε = h, t = 0.1, ν = 0.01x255 2, λ L = λ T = 5. (a) Initial level set contours. (b) Level set contours after 50 interations. (c) Final level set contours after 117 iterations. Figure 5.7. Frame 1 results. Final contour evolution and connected components labeling for frame 1. (a) Background pixels from original image. (b) Background (blue) v. objects (red). (c) Background (blue) v. individual objects (colored). to the connected components labeling function to produce Figure 5.7c with each color corresponding to a distinct cell. We show that there are three distinct cells in the image. As shown in Figure 5.7c, the labeling and counting function is not dependent on a predefined cell size. The cell regions are separated and counted regardless of their size. Here, we have a cell that has completed cell division (two small cells to the left) and one cell that has not divided. The same process was applied to frame 6. The segmentation process is displayed in Figure 5.8. We show that the stopping criterion was met and contour evolution was stopped after 129 iterations. Our method accurately extracts the cell regions from the image. The results of the labeling function (Figure 5.9) show the ability of our method to deal with dividing cells. Despite the cell walls pinching in as cell division begins in the right hand cell, the method correctly labels only three distinct cells in the cell region. The segmentation results for the initial cells show that our method deals with the complex topologies that arise during and after cell division in E. coli. In addition, the labeling function is able to correctly identify unique cells and accurately connect related cell pixels.

37 29 Figure 5.8. Evolving contours overlaid on original frame 6. Type 3 initialization with h = x = y = 1, ε = h, t = 0.1, ν = 0.01x255 2, λ L = λ T = 5. (a) Initial level set contours. (b) Level set contours after 50 interations. (c) Final level set contours after 129 iterations. Figure 5.9. Frame 6 results. Final contour evolution and connected components labeling for frame 6. (a) Background pixels from original image. (b) Background (blue) v. objects (red). (c) Background (blue) v. individual objects (colored). The large number of iterations required to produce results for frame 6 indicated that later frames would likely require an increasing number of iterations for convergence to the solution. 5.5 COMPLEX DETECTION: LARGE POPULATION OF CELLS We apply our modified multiphase level set method to frames 29 and 62 of the E. coli cell division video (Figure 5.10) [14] and show the robustness of our method in the presence of a large, congested cell mass. We evaluate the performance using type 3 initialization for frame 29. To illustrate the non-reliance on initialization, we use type 2 initialization for frame 62. We show that our method will extract the cell boundaries, even in a congested image, regardless of the choice of initialization. We use ν = x 255 2, to capture the fine detail of the cells, and we let λ L = 2 and λ T = 5 to avoid stopping before all cell boundaries are detected.

38 30 Figure Complex video frames. These later frames contain numerous E. coli cells as the result of cell division. (a) Frame 29. (b) Frame 62. Source: UNK, Bacteria growth video. =gewzdydciwc, accessed February 2012, July The reduced value of ν allowed the method to deal with the large cell mass that developed between frame 6 and frame 29. The more sensitive ν is critical when the cell mass get as large as that in frame 62 and larger in later frames. The method identifies the true cell boundaries after only 89 iterations for frame 29 (Figure 5.11). The robustness of the labeling function is displayed with these results. We easily separate the background from the cell region using the results of φ 89 1 and φ 89 2 as shown in Figures 5.12a and 5.12b. In Figure 5.12a we see the background (original image) vs. the cell region (white). In Figure 5.12b, the blue pixels represent background and the red pixels represent the cell region. These results are provided to the connected components labeling function to produce Figure 5.12c with each color corresponding to a distinct cell. Despite the unpredictable growth rate and direction, it accurately separates the distinct cell regions. The method does not falsely separate cells undergoing cell division or those stretching out in preparation for cell division. We automatically identify ten distinct cells, as shown by the color coded results. This is verified by manually counting the cells in the original image (Figure 5.10a). Figure Evolving contours overlaid on original frame 29. Type 3 initialization with h = x = y = 1, ε = h, t = 0.1, ν = x 255 2, λ L = 2, λ T = 5. (a) Initial level set contours. (b) Level set contours after 50 interations. (c) Final level set contours after 89 iterations.

39 31 Figure Frame 29 results. Final contour evolution and connected components labeling for frame 29. (a) Background pixels from original image. (b) Background (blue) v. objects (red). (c) Background (blue) v. individual objects (colored). We finally consider a complicated case with numerous cells in the image space. It is clear that manual segmentation would be an arduous task in this frame, but especially in frames to follow. The cell mass is growing rapidly and a manual process will become error prone. The use of type 2 initialization (Figure 5.13a) illustrates the ability of the method to identify cell boundaries regardless of the initial contour. We use ν = x 255 2, λ L = 2, and λ T = 5 to ensure the fine detail of the cells is appropriately handled. The cell size inconsistency and the use of type 2 initialization will require more iterations for all the cell boundaries to be identified. The two contours slowly move toward and around the cell region, separating it from the background and bright pixels (Figure 5.13b). After 550 iterations, the evolving curves successfully surround the cell region and the method is stopped (Figure 5.13c). Figure Evolving contours overlaid on original frame 62. Type 2 initialization with h = x = y = 1, ε = h, t = 0.1, ν = x 255 2, λ L = 2, λ T = 5. (a) Initial level set contours. (b) Level set contours after 50 interations. (c) Final level set contours after 550 iterations. Figure 5.14 shows the results of the method and labeling function. The method shows some sensitivity to dark pixels in the background region (Figure 5.14). The labeling function conducts a filtering step to eliminate stray cell regions that are disconnected from the main

40 32 Figure Frame 62 results. Final contour evolution and connected components labeling for frame 62. (a) Background pixels from original image. (b) Background (blue) v. objects (red). (c) Background (blue) v. individual objects (colored). cell mass. However, the soda straw point of view in microscopy video often results in cells entering or exiting the field of view. In addition, while cells typically stay close to the cell mass, they have the ability to break off and roam free. To avoid falsely removing a separated cell mass, there is a size threshold for the filter. Only regions (away from the cell mass) that are less than twenty pixels will be filtered out of the cell region and labeled as background. We also see that a cluster of four distinct cells at the center of the cell mass is not separated by the evolving contours. Additional iterations could separate these cells at the expense of downgrading the contour edges extracting other cells in the cell region. Even in the original image it is difficult to manually distinguish the separate cells in this four cell cluster. Despite the method not separating this small cluster of cells, it handles the remainder of the image well. The method identifies fifty-four distinct cells in the image (Figure 5.14). Though there are some errors in the labeling, it is important to point out how the method handled the two large, non-divided cells on the left as well as the smaller cells at the top. The labeling function is able to correctly identify unique cells and connect related cell pixels regardless of the direction or size that they grow. The accuracy of the labeling function is directly dependent on the accuracy of the level set method. If two distinct cells are not separated by the evolving curves, then the labeling function will treat them as one cell. Improvements could be made to the labeling function when dealing with large numbers of cells. Specifically, there is a need for a more complex, second labeling round. This round could compare the detected cells to known cell shapes and eliminate false connections made based on proximity. The results presented suggest that a single phase level set method could perform well to identify the cell boundaries in the video frames. In the provided output, it appears that both level set functions are extracting the same image regions. To test this, we superimpose the two initial contours (Figure 5.15a), replicating a single phase level set. We show that two distinct

41 33 Figure Frame 62 with superimposed initial contours. (a) Superimposed initial level set contours. (b) Level set contours after 25 iterations. (c) Level set contours after 113 iterations. initial contours are required for individual cell detection. As is evident in the previous figures, the cells have a glow of bright pixels around them. This white halo surrounding the dark cell soma is characteristic of E. coli cells in phase-contrast microscopy video. The contrast between the background pixels and the cell regions is less extreme than the contrast between the background pixels and the bright pixels. As such, the single level set method is sensitive to the bright pixels around and between the cells. The single level set method focuses on the bright areas rather than the individual cells. At first, it evolves toward the bright pixels between the cells (Figure 5.15b). The averaging step, c I, forces the level set to move out and settle around the cell mass as a whole, where there is a balance of bright and dark pixels (Figure 5.15c). We show the four distinct image regions identified by the four phase contour evolution process (Figure 5.16). This helps distinguish the difference in the two contours and the results of the evolution. The four phase level set method deals with these bright pixels and extracts the cells, bright pixels, and background as separate regions, which the single level set function was unable to do.

42 Figure Frame 62 binary output. Illustrates final four regions as defined by the level set contours (frame 62). Top left: φ 1 > 0 and φ 2 < 0 Top Right: φ 1 < 0 and φ 2 > 0 Bottom Left: φ 1 > 0 and φ 2 > 0 Bottom Right: φ 1 < 0 and φ 2 < 0. 34