MEDICAL IMAGE COMPUTING (CAP 5937) LECTURE 10: Medical Image Segmentation as an Energy Minimization Problem

Size: px

Start display at page:

Download "MEDICAL IMAGE COMPUTING (CAP 5937) LECTURE 10: Medical Image Segmentation as an Energy Minimization Problem"

Chrystal Skinner
5 years ago
Views:

1 SPRING 07 MEDICAL IMAGE COMPUTING (CAP 97) LECTURE 0: Medical Image Segmentation as an Energy Minimization Problem Dr. Ulas Bagci HEC, Center for Research in Computer Vision (CRCV), University of Central Florida (UCF), Orlando, FL 84. or

2 Energy functional Outline Data and Smoothness terms Graph Cut Min cut Max Flow Applications in Radiology Images

3 Motivation Manual annotation through expert raters. Shown are image patches with the tumor structures that are annotated in the different modalities (top left) and the final labels for the whole dataset (right). Image patches show from left to right: the whole tumor visible in FLAIR (A), the tumor core visible in T (B), the enhancing tumor structures visible in Tc (blue), surrounding the cystic/necrotic components of the core (green) (C). Segmentations are combined to generate the final labels of the tumor structures (D): edema (yellow), non-enhancing solid core (red), necrotic/cystic core (green), enhancing core(blue). Credit: BRATS paper/tmi

4 4 Labeling & Segmentation Labeling is a common way for modeling various computer vision problems (e.g. optical flow, image segmentation, stereo matching, etc)

5 Labeling & Segmentation Labeling is a common way for modeling various computer vision problems (e.g. optical flow, image segmentation, stereo matching, etc) The set of labels can be discrete (as in image segmentation) L = {l,...,l m } with L = m

6 6 Labeling & Segmentation Labeling is a common way for modeling various computer vision problems (e.g. optical flow, image segmentation, stereo matching, etc) The set of labels can be discrete (as in image segmentation) L = {l,...,l m } with L = m Or continuous (tracking, etc.) L R n for n

7 7 Labeling is a function Labels are assigned to sites (pixel locations)

8 8 Labeling is a function Labels are assigned to sites (pixel locations) For a given image, we have = N cols.n rows

9 9 Labeling is a function Labels are assigned to sites (pixel locations) For a given image, we have = N cols.n rows Identifying a labeling function (with segmentation) is f :! L

10 0 Labeling is a function Labels are assigned to sites (pixel locations) For a given image, we have Identifying a labeling function (with segmentation) is f :! L = N cols.n rows We aim at calculating a labeling function that minimizes a given (total) error or energy

11 Labeling is a function Labels are assigned to sites (pixel locations) For a given image, we have Identifying a labeling function (with segmentation) is f :! L = N cols.n rows We aim at calculating a labeling function that minimizes a given (total) error or energy E(f) = X p [E data (p, f p )+ X qa(p) E smooth (f p,f q )] *A is an adjacency relation between pixel locations

12 Example of Energy Minimization Methods Credit: Zhao et al., IJNMBE

13 Energy Function? Penalizing results which are not compatible with the observed images/volumes

14 4 Energy Function? Penalizing results which are not compatible with the observed images/volumes Unary (data) cost (inverted)

15 Energy Function? Penalizing results which are not compatible with the observed images/volumes Enforcing spatial coherence. Pairwise (boundary) cost (inverted)

16 6 Energy Function? Penalizing results which are not compatible with the observed images/volumes Enforcing spatial coherence. Pairwise (boundary) cost (inverted) p q

17 7 Segmentation as an Energy Minimization Problem E data assigns non-negative penalties to a pixel location p when assigning a label to this location. 0//

18 8 Segmentation as an Energy Minimization Problem E data assigns non-negative penalties to a pixel location p when assigning a label to this location. E smooth assigns non-negative penalties by comparing the assigned labels f p and f q at adjacent positions p and q.

E smooth assigns non-negative penalties by comparing the assigned labels f p and

19 9 Segmentation as an Energy Minimization Problem E data assigns non-negative penalties to a pixel location p when assigning a label to this location. E smooth assigns non-negative penalties by comparing the assigned labels f p and f q at adjacent positions p and q. p Edge q Node/vertex pixel Image as a graph representation

20 0 Segmentation as an Energy Minimization Problem E data assigns non-negative penalties to a pixel location p when assigning a label to this location. E smooth assigns non-negative penalties by comparing the assigned labels f p and f q at adjacent positions p and q. This optimization model is characterized by local interactions along edges between adjacent pixels, and often called MRF (Markov Random Field) model.

21 E(f) = X p Energy Function-Details [E data (p, f p )+ X qa(p) E smooth (f p,f q )]

22 E(f) = X p Energy Function-Details [E data (p, f p )+ X qa(p) E smooth (f p,f q )] Example Data Term: E data (p, f p )= (p) (p = 0) = logp(p BG) (p = ) = logp(p FG)

23 E(f) = X p Example Data Term: Example Smoothness Term: Energy Function-Details [E data (p, f p )+ X qa(p) E smooth (f p,f q )] E data (p, f p )= (p) (p = 0) = logp(p BG) (p = ) = logp(p FG) (p, q) =K pq (p 6= q) where K pq = exp( (I p I q ) /( )) p, q

24 4 E(f) = X p Energy Function-Details [E data (p, f p )+ X qa(p) E smooth (f p,f q )] E(p) = X p p(p)+ X p X pq(p, q) qa(p) p = argmin pl E(p)

25 E(f) = X p Energy Function-Details [E data (p, f p )+ X qa(p) E smooth (f p,f q )] E(p) = X p p(p)+ X p X pq(p, q) qa(p) p = argmin pl E(p) To solve this problem, transform the energy functional into mincut/max-flow problem and solve it!

26 6 Graph Cuts for Optimal Boundary Detection (Boykov ICCV 00) Binary label: foreground vs. background User labels some pixels Exploit Statistics of known Fg & Bg Smoothness of label Turn into discrete graph optimization Graph cut (min cut / max flow) F F F F F F B B B B B 6

27 7 Graph Cut Each pixel is connected to its neighbors in an undirected graph Goal: split nodes into two sets A and B based on pixel values, and try to classify Neighbors in the same way too!

28 8 Graph Cut Each pixel is connected to its neighbors in an undirected graph Goal: split nodes into two sets A and B based on pixel values, and try to classify Neighbors in the same way too!

29 9 Each pixel = node Graph-Cut Add two nodes F & B Labeling: link each pixel to either F or B F Desired result F F F F F B B B B B

30 0 Cost Function: Data term Put one edge between each pixel and both F & G Weight of edge = minus data term F B

31 Cost Function: Smoothness term Add an edge between each neighbor pair Weight = smoothness term F B

32 Min-Cut Energy optimization equivalent to graph min cut Cut: remove edges to disconnect F from B Minimum: minimize sum of cut edge weight F cut B

33 Min-Cut Source 9 v v 4 Graph (V, E, C) Vertices V = {v, v... v n } Edges E = {(v, v )...} Costs C = {c (, )...} Sink

34 4 Min-Cut What is a st-cut? Source An st-cut (S,T) divides the nodes between source and sink. 9 v v 4 What is the cost of a st-cut? Sum of cost of all edges going from S to T Sink = 6

35 Min-Cut What is a st-cut? Source 9 v v Sink = 7 An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost

36 6 How to compute min-cut? Source 9 v v Sink 4 Solve the dual maximum flow problem Compute the maximum flow between Source and Sink Constraints Edges: Flow < Capacity Nodes: Flow in & Flow out Min-cut\Max-flow Theorem In every network, the maximum flow equals the cost of the st-mincut

37 7 Flow = 0 Source Max-Flow Algorithms Augmenting Path Based Algorithms 9 v v Sink 4. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

38 8 Flow = 0 Source Max-Flow Algorithms Augmenting Path Based Algorithms 9 v v Sink 4. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

39 9 Flow = 0 + Source Max-Flow Algorithms Augmenting Path Based Algorithms - 9 v v - Sink 4. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

40 40 Flow = Source Max-Flow Algorithms Augmenting Path Based Algorithms 0 9 v v Sink 4. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

41 4 Flow = Source Max-Flow Algorithms Augmenting Path Based Algorithms 0 9 v v Sink 4. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

42 4 Flow = Source Max-Flow Algorithms Augmenting Path Based Algorithms 0 9 v v Sink 4. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

43 4 Flow = + 4 Source Max-Flow Algorithms Augmenting Path Based Algorithms 0 v v Sink 0. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

44 44 Flow = 6 Source Max-Flow Algorithms Augmenting Path Based Algorithms 0 v v Sink 0. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

45 4 Flow = 6 Source Max-Flow Algorithms Augmenting Path Based Algorithms 0 v v Sink 0. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

46 46 Flow = 6 + Source Max-Flow Algorithms Augmenting Path Based Algorithms v v + Sink 0. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

47 47 Flow = 7 Source Max-Flow Algorithms Augmenting Path Based Algorithms v v Sink 0. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

48 48 Flow = 7 Source Max-Flow Algorithms Augmenting Path Based Algorithms v v Sink 0. Find path from source to sink with positive capacity. Push maximum possible flow through this path. Repeat until no path can be found Algorithms assume non-negative capacity

49 49 Another Example-Max Flow source 9 Ford & Fulkerson algorithm (96) 6 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 8 Find the path in the residual graph sink End

50 0 Another Example-Max Flow source 9 Ford & Fulkerson algorithm (96) 6 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 8 Find the path in the residual graph sink End

51 Another Example-Max Flow source 9 Ford & Fulkerson algorithm (96) 6 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 8 Find the path in the residual graph sink End flow =

52 Another Example-Max Flow - source 9 Ford & Fulkerson algorithm (96) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 8- Find the path in the residual graph sink End flow =

53 Another Example-Max Flow source 9 Ford & Fulkerson algorithm (96) 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow =

54 4 Another Example-Max Flow source 9 Ford & Fulkerson algorithm (96) 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow =

55 Another Example-Max Flow source 9 Ford & Fulkerson algorithm (96) 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow = 6

56 6 6 Another Example-Max Flow source 4 sink Ford & Fulkerson algorithm (96) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) End Find the path in the residual graph flow = 6

57 7 Another Example-Max Flow source 6 Ford & Fulkerson algorithm (96) 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow = 6

58 8 Another Example-Max Flow source 6 Ford & Fulkerson algorithm (96) 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow = 6

59 9 Another Example-Max Flow source 6 Ford & Fulkerson algorithm (96) 4 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow =

60 60 6 Another Example-Max Flow source sink Ford & Fulkerson algorithm (96) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) End Find the path in the residual graph flow =

61 6 6 Another Example-Max Flow source 4 sink 7 6 Ford & Fulkerson algorithm (96) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) End Find the path in the residual graph flow =

62 6 6 Another Example-Max Flow source 4 sink 7 6 Ford & Fulkerson algorithm (96) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) End Find the path in the residual graph flow =

63 6 6 Another Example-Max Flow source 4 sink 7 6 Ford & Fulkerson algorithm (96) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) End Find the path in the residual graph flow =

64 64 Another Example-Max Flow - source Ford & Fulkerson algorithm (96) 4 7 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 - Find the path in the residual graph sink End flow =

65 6 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 4 7 Find the path from source to sink 6 While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow =

66 66 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 4 7 Find the path from source to sink 6 While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow =

67 67 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 4 7 Find the path from source to sink 6 While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6 Find the path in the residual graph sink End flow =

68 68 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 4-7 Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) 6- Find the path in the residual graph sink End flow =

69 69 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 7 Find the path from source to sink 4 6 While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) Find the path in the residual graph sink End flow =

70 70 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 7 Find the path from source to sink 4 6 While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) Find the path in the residual graph sink End flow =

71 7 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 7 Find the path from source to sink 4 6 While (path exists) flow += maximum capacity in the path Build the residual graph ( subtract the flow) Find the path in the residual graph sink End flow =

72 7 Another Example-Max Flow source Ford & Fulkerson algorithm (96) 7 Why is the solution globally optimal? 6 4 sink flow =

73 7 S Another Example-Max Flow source Ford & Fulkerson algorithm (96) 7 Why is the solution globally optimal? 6. Let S be the set of reachable nodes in the residual graph 4 sink flow =

74 74 Another Example-Max Flow S source sink Ford & Fulkerson algorithm (96) Why is the solution globally optimal?. Let S be the set of reachable nodes in the residual graph. The flow from S to V - S equals to the sum of capacities from S to V S

75 7 / / / 0/ /6 0/ 0/ Another Example-Max Flow source /6 8/8 sink /4 0/ flow = 8/9 / 0/ / / / / Individual flows obtained by summing up all paths Ford & Fulkerson algorithm (96) Why is the solution globally optimal?. Let S be the set of reachable nodes in the residual graph. The flow from S to V - S equals to the sum of capacities from S to V S. The flow from any A to V - A is upper bounded by the sum of capacities from A to V A 4. The solution is globally optimal

76 Another Example-Max Flow 76 source sink S T cost = 8 source sink S T

77 77 Another Example-Max Flow S source cost = 6 8 sink T

78 78 Another Example-Max Flow source C(x) = x + 9x + 4x + x (-x ) + x (-x ) 9 + x (-x ) + x (-x ) + x (-x ) + x 4 (-x ) + x (-x ) + 6x (-x ) + x 6 (-x ) + x (-x 6 ) x 4 x + x 6 (-x ) + x 4 (-x ) + x 6 (-x ) + 6(-x 4 ) x + 8(-x ) + (-x 6 ) x 4 6 x 6 x 6 8 sink

79 79 Another Example-Max Flow source C(x) = x + 9x + 4x + x (-x ) 9 + x (-x ) + x (-x ) + x (-x ) + x 4 (-x ) + x (-x ) + x (-x ) + x 6 (-x ) + x (-x 6 ) x 4 x + x 6 (-x ) + x 4 (-x ) + x 6 (-x ) + 6(-x 4 ) x + (-x ) + (-x 6 ) x x + x (-x ) + x (-x ) + (-x ) x 6 x 6 8 sink

80 80 Another Example-Max Flow source C(x) = x + 9x + 4x + x (-x ) 9 + x (-x ) + x (-x ) + x (-x ) + x 4 (-x ) + x (-x ) + x (-x ) + x 6 (-x ) + x (-x 6 ) x 4 x + x 6 (-x ) + x 4 (-x ) + x 6 (-x ) + 6(-x 4 ) x + (-x ) + (-x 6 ) x x + x (-x ) + x (-x ) + (-x ) x 6 x 6 8 sink x + x (-x ) + x (-x ) + (-x ) = + x (-x ) + x (-x )

81 8 Another Example-Max Flow source C(x) = + x + 6x + 4x + x (-x ) 9 + x (-x ) + x (-x ) + x (-x ) + x 4 (-x ) + x (-x ) + x (-x ) + x 6 (-x ) + x (-x 6 ) x 4 x + x (-x 4 ) + 6(-x 4 ) x + (-x ) + (-x 6 ) + x (-x ) x 4 + x + x 6 (-x ) + x (-x 6 ) + (-x ) x 6 x 6 sink x + x 6 (-x ) + x (-x 6 ) + (-x ) = + x (-x 6 ) + x 6 (-x )

82 8 source 6 x 4 x x 4 Another Example-Max Flow x x 6 x C(x) = 6 + x + 6x + 4x + x (-x ) + x (-x ) + x (-x ) + x (-x ) + x 4 (-x ) + x (-x ) + x (-x ) + x 6 (-x ) + x (-x 6 ) + x (-x 4 ) + 6(-x 4 ) + x (-x 6 ) + x 6 (-x ) + (-x ) + (-x 6 ) + x (-x ) 6 sink

83 Another Example-Max Flow 8 min cut source C(x) = + x + 4x + x (-x ) + x (-x ) + 7x (-x ) + x (-x 4 ) + x (-x ) + 6x (-x 6 ) + 6x (-x ) x x 7 + x (-x 4 ) + 4(-x 4 ) + x (-x 6 ) + x 6 (-x ) x 6 x 4 cost = 0 x 6 cost = 0 x S 4 sink T S set of reachable nodes from s All coefficients positive Must be global minimum

84 84 History of Max-Flow Algorithms Augmenting Path and Push-Relabel n: #nodes m: #edges U: maximum edge weight Algorithms assume nonnegative edge weights

85 Software Packages for Optimization 8

86 Applications Used in Energy Minimization Based Segmentation Methods 86

87 Applications 87

88 88 Interactive Organ Segmentation (Boykov and Jolly, MICCAI 000) Segmentation of multiple objects. (a-c): Cardiac MRI. (d): Kidney CE- MR angiography

89 89 Interactive Organ Segmentation (Boykov and Jolly, MICCAI 000) Segmentation of the right lung in CT. (a): representative D slice of original D data. (b): segmentation results on the slice in (a). (c-d) D visualization of segmentation results.

90 90 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily visualized despite the proximity of surrounding organs such as the ribs and pulmonary blood vessels.

91 9 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily visualized despite the proximity of surrounding organs such as the ribs and pulmonary blood vessels. Numerous techniques have been described for segmenting the left ventricle of the heart in images from various types of medical scanners but rarely has the entire heart been segmented.

92 9 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily visualized despite the proximity of surrounding organs such as the ribs and pulmonary blood vessels. Numerous techniques have been described for segmenting the left ventricle of the heart in images from various types of medical scanners but rarely has the entire heart been segmented. Graph-cut formation:

93 9 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily visualized despite the proximity of surrounding organs such as the ribs and pulmonary blood vessels. Numerous techniques have been described for segmenting the left ventricle of the heart in images from various types of medical scanners but rarely has the entire heart been segmented. Graph-cut formation:

94 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily

94 94 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily visualized despite the proximity of surrounding organs such as the ribs and pulmonary blood vessels. Numerous techniques have been described for segmenting the left ventricle of the heart in images from various types of medical scanners but rarely has the entire heart been segmented. Graph-cut formation: (C is center)

95 9 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Isolating the entire heart allows the coronary vessels on the surface of the heart to be easily visualized despite the proximity of surrounding organs such as the ribs and pulmonary blood vessels. Numerous techniques have been described for segmenting the left ventricle of the heart in images from various types of medical scanners but rarely has the entire heart been segmented. Graph-cut formation: (C is center) If cos(.) <0, (0 otherwise.)

96 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Top Left: A balloon is expanded within the heart.

96 96 AUTOMATIC HEART ISOLATION FOR CT CORONARY VISUALIZATION USING GRAPH-CUTS (Funka-Lea et al, ISBI 006) Top Left: A balloon is expanded within the heart. The heart wall pushes the balloon toward the heart center as the balloon grows. Top right: volume rendering of original heart volume. Bottom left: heart cropped based on segmentation mask. Bottom right: volume rendering after automatic heart isolation algorithm.

97 97 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) Proposed a method for simultaneously segmenting longitudinal magnetic resonance (MR) images

98 98 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) Proposed a method for simultaneously segmenting longitudinal magnetic resonance (MR) images D MRI + time component (longitudinal)

99 99 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) Proposed a method for simultaneously segmenting longitudinal magnetic resonance (MR) images D MRI + time component (longitudinal) A 4D graph is used to represent the longitudinal data: edges are weighted based on spatial and intensity priors and connect spatially and temporally neighboring voxels represented by vertices in the graph.

100 00 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) Proposed a method for simultaneously segmenting longitudinal magnetic resonance (MR) images D MRI + time component (longitudinal) A 4D graph is used to represent the longitudinal data: edges are weighted based on spatial and intensity priors and connect spatially and temporally neighboring voxels represented by vertices in the graph. Solving the min-cut/max-flow problem on this graph yields the segmentation for all time-points in a single step

101 0 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) Proposed a method for simultaneously segmenting longitudinal magnetic resonance (MR) images D MRI + time component (longitudinal) A 4D graph is used to represent the longitudinal data: edges are weighted based on spatial and intensity priors and connect spatially and temporally neighboring voxels represented by vertices in the graph. Solving the min-cut/max-flow problem on this graph yields the segmentation for all time-points in a single step Time-series image can be considered as a single (4D) image, with the following dimensions: x,y,z and t

102 0 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) Proposed a method for simultaneously segmenting longitudinal magnetic resonance (MR) images D MRI + time component (longitudinal) A 4D graph is used to represent the longitudinal data: edges are weighted based on spatial and intensity priors and connect spatially and temporally neighboring voxels represented by vertices in the graph. Solving the min-cut/max-flow problem on this graph yields the segmentation for all time-points in a single step Time-series image can be considered as a single (4D) image, with the following dimensions: x,y,z and t 6 spatial neighbors in D And temporal neighbors and

103 0 Measurement of hippocampal atrophy using 4D graph-cut segmentation: Application to ADNI (Wolz, et al. NeuroImage 0) a. Right hippocampus segmentation (baseline), b. follow up segmentation ( months)

104 04 Integrated Graph Cuts for Brain Image Segmentation (Song et al, MICCAI 006) In addition to image intensity, tissue priors and local boundary information are integrated into the edge weight metrics in the graph.

105 0 Integrated Graph Cuts for Brain Image Segmentation (Song et al, MICCAI 006) In addition to image intensity, tissue priors and local boundary information are integrated into the edge weight metrics in the graph. Example of the graph with three terminals for brain MRI tissue segmentation of gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF). The set of nodes V includes all voxels and terminals. The set of edges E includes all n-links and t-links. n-links: voxel-to-voxel edges t-links: voxel-to-terminal edges

106 Integrated Graph Cuts for Brain Image Segmentation (Song et al, MICCAI 006) 06

107 07 Integrated Graph Cuts for Brain Image Segmentation (Song et al, MICCAI 006) Regularization term Data term Atlas term Pairwise term

108 08 Integrated Graph Cuts for Brain Image Segmentation (Song et al, MICCAI 006) Regularization term Data term Atlas term (T, segmentation results)

109 GC + Appearance Model 09

110 GC + Appearance Model 0

111 GC + Shape Model

Regions used for calculating Dice score, sensitivity, specificity, and robust Hausdorff score. Region T is the true lesion area (outline blue), T 0 is the remaining normal area.

112 Regions used for calculating Dice score, sensitivity, specificity, and robust Hausdorff score. Region T is the true lesion area (outline blue), T 0 is the remaining normal area. P is the area that is predicted to be lesion by for example an algorithm (outlined red), and P 0 is predicted to be normal. P has some overlap with T in the right lateral part of the lesion, corresponding to the area referred to as P Λ T in the definition of the Dice score. (Credits: BRATS paper)

113 Summary Data and Smoothness Terms -> Graph based segmentation methods Additional terms can(should) be added into segmentation formulation based observation/need and problem definition Problems formulated as a MRF task can be solved by max-flow/mincut

114 4 Slide Credits and References Fredo Durand M. Tappen R. Szelisky J.Malcolm, Graph Cut in Tensor Scale Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D images. Yuri Boykov and Marie-Pierre Jolly. In International Conference on Computer Vision, (ICCV), vol. I, A Comparative Study of Energy Minimization Methods for Markov Random Fields. Rick Szeliski, Ramin Zabih, Daniel Scharstein, Olga Veksler, Vladimir Kolmogorov, Aseem Agarwala, Marshall Tappen, Carsten Rother. ECCV P. Kumar, Oxford University.

MEDICAL IMAGE COMPUTING (CAP 5937) LECTURE 10: Medical Image Segmentation as an Energy Minimization Problem

MEDICAL IMAGE COMPUTING (CAP 5937) LECTURE 10: Medical Image Segmentation as an Energy Minimization Problem SPRING 06 MEDICAL IMAGE COMPUTING (CAP 97) LECTURE 0: Medical Image Segmentation as an Energy Minimization Problem Dr. Ulas Bagci HEC, Center for Research in Computer Vision (CRCV), University of Central