Graph-cut methods for joint reconstruction and segmentation

Transcription

1 Graph-cut methods for joint reconstruction and segmentation W. Clem Karl Department of Electrical and Computer Engineering Department of Biomedical Engineering Boston University Joint work with A. uysuzoglu, D. Castanon

2 Outline Discrete amplitude inverse problems Use of graph cuts for discrete problems Graph-cuts and inverse problems he inverse problem messes things up New ideas: Idea 1: Separable approximation approach Idea 2: Understand structure Graph representable class of inverse problems Idea 3: Exploit structure Graph-cut based dual decomposition method for inverse problems Examples 2

3 Overview Inverse problems: Medical tomography images from projections Scanning electron tomography of materials grains Security scanning imaging inside luggage Deblurring, Super-resolution fusion and deblurring Discrete-amplitude inverse problems Medical tomography bone, tissue, interventional instrument Material science grain volume fractions Security threat classification Deblurring segmentation 3

4 Examples Electron omography for Material Science: Sample omographic Data X-Ray Source Sinogram Segmented Reconstruction Deblurring (coded aperture flutter shutter ) 4 Images from M. DeGraef, omography in Materials Science Workshop, Dayton, OH, Dec , 21 Images from Coded Exposure Photography, R. Raskar, A. Agrawal, J. umblin, ACM Siggraph 26 Image Camera System Blurred Image Segmented Reconstruction

5 Common problem characteristics Indirect or coded observation y of latent variables x Few amplitudes are of interest: x discrete Scene structure is important Focus on locations with specific values where stuff is Data is limited, information poor Ad hoc methods perform poorly 5

6 Problem Formulation Observations: y Cx n Data Vector Imaging Operator Discrete Amplitude Image Noise Estimate discrete level image x by cost minimization: ˆx argmin x Disc Amp J(x) argmin x Disc Amp J data (x) J prior (x) NP Hard! Data Fidelity: Prior (MRFs, V, sparse, etc): J data J prior ( x) y Cx ( x) Dx

7 Approach #1 wo-step decoupled process Continuous relaxation 1. Relax discrete assumption on x 2. Solve resulting continuous inverse problem (e.g. with V) 3. Segment continuous reconstruction result Ad hoc, wastes information argmin x Disc Amp 2 2 y Cx 2 J prior (x) argmin y Cx 2 x n J prior (x) 7

8 Approach #2: DAR (Discrete AR) Algorithm for discrete tomography Idea: Interior regions are reliable DAR Algorithm flow chart Free pixels are boundary + random pixels No explicit regularization Discretization decoupled from reconstruction Very popular in materials science Iterative variant of Approach #1 Initialize with SAR Segment Identify boundary pixels as free Apply SAR to free pixels Smooth Image 8

9 Approach #3: Graph-cut Methods Direct solution of underlying discrete problem All start by representing problem on a graph Powerful and efficient methods for these problems exist Some nice performance results Global optimality! Efficient algorithms Min-cut/Max-flow Polynominal time 9

10 Graph Cuts when C=I, Binary Case argmin y x 2 2 J prior (x) argmin Boston x,1 University Slideshow itle Goes xhere,1 p (x p ) p,q (x p, x q ) p Unary erms p,q Pairwise erms Segmentation case p Vertices are pixel values Edges are costs Graph representation Submodularity of pairwise terms Submodular when C = I since pairwise terms arise from smoothness prior (,) p, q (1,1) p, q (,1) p, p, q q (1,) q 1

11 Graph Cuts when C=I, Multi-label case 2 argmin y x Boston University 2 J prior (x) argmin p (x p ) p,q (x p, x q ) xl n Slideshow itle Goes xl Here n p p,q Pixel values from set L n Unary erms Pairwise erms Can t exploit binary graph-cut machinery Multi-label extensions exist, but clumsy ypically break problem into sequence of binary problems - swap iteratively minimize wrt pairs of labels expansion iteratively minimize all labels wrt label Only obtain approximate solutions but good behavior in practice Sub-problems still need to be submodular! Still true when C = I. hings work well in this case p q 11

12 Graph Cuts when C I (Inversion) argmin x Disc Amp y Cx 2 2 J prior (x) argmin Overall cost is not submodular (graph representable) Conventional graph-cut approaches do not work Problem is with data term: x Disc Amp p p (x p ) p,q (x p, x q ) Unary erms p,q Pairwise erms J data ( x) 2 y Cx y y 2y 2 Cx x C Cx Contains general cross terms involving x p x q [Raj `5] Coupling of pixels due to non-diagonal C makes J data non-graph-representable 12

13 Related work on graph-cuts, inversion Graph Cuts Boston University [Raj `5] Slideshow : Deconvolution itle Goes Here Approximate non-submodular cross-terms in J data Replace non-submodular cross-terms C pq x p x q with linear terms C pq [x () p x q + x p x () q ] Method is for - swap algorithm Assumes non-negative C [Raj `6, Rother `7] : MRI Reconstruction Use of roof-duality to remap some non-submodular terms (QPBO) Cannot always be done Can leave pixels unlabeled [Cremers 6] : Learned submodular models Inverse Problems [Wright, Nowak, Figueiredo `9] : Sparse reconstruction Continuous amplitude problems (not focused on graph cuts) Separable approximation of data term Discrete omography [Batenburg, `11]: DAR Ad hoc iterative thresholding method 13

14 Idea #1: Use separable approximation of J data Boston Relax University discrete Slideshow itle assumption Goes Here on x Perform aylor expansion of J data (x) around x Approximate Hessian by scaled diagonal Separable approximation eliminates cross-pixel coupling (Wright `9) J data ( x) 2 y Cx y y 2y 2 Cx x C Cx Ĵ data (x x,) J data (x )J data (x ) (x x ) 1 2 (x x ) D(x x ) J data ( x ) 2C D diag(2c y 2C C) diag( Approximation Parameter Cx 2 J data ) Diagonal of Hessian diag[c C] 14

15 Overall new approach Replace original cost with surrogate cost J ( x) J ( x) J ( x) data prior Not Graph Representable Jˆ( x) Jˆ data ( x x, ) J prior ( x) Graph Representable Surrogate cost is graph representable (submodular) Any multi-label graph-cut method can now minimize -expansion in this work Iteratively minimize surrogate cost 15

16 Graph Cuts w/separable Approximation (GCWSA) GCWSA Algorithm: 1. Initialize: x () 2. Use graph-cut algorithm to minimize surrogate cost x ( k 1) argmin x Disc Amp Jˆ data ( x x ( k ), ) J k prior ( x) 3. k Update (Acceptance test): a. If J(x (k+1) )>J(x (k) ) increase k and redo 2. (forces smaller step size) b. If J(x (k+1) )<J(x (k) ) accept updated estimate, set k k+1 and goto 2. 16

17 Efficient computation for relaxation parameter k Ĵ data (x x,) J data (x )J data (x ) (x x ) 1 2 (x x ) D(x x ) Good choice of improves solution Repeated computation for different Observation: Changing k only affects unary terms (via diagonal approx) Pairwise terms remain unchanged (many of these) Flow Recycling [Kohli, 25] Reuse min-flow results Modify previous residual graph Most terms remain unchanged Order of magnitude faster than naïve way! Diagonal of C C scaled by p 17

18 Experiment: Noisy Multi-level 256 x 256 Image Known Intensity Levels {, 5, 1}. Projections over 12 degree angular span 1 degree separation between projections 362 detector values per angle 3 db AWGN Ground truth image 18

19 Experiment: Noisy Multi-level rue FBP Non-neg SAR Unregularized GCWSA DAR Segmented FBP Segmented SAR Regularized GCWSA Streaks persist 19

20 Reconstruction Errors of Different Methods Boston Mislabeled University Slideshow Pixels: itle Goes Here Multi-label Phantom Noiseless 3dB Noise 1 Log scale! FBP SAR DAR Ureg GCWSA Reg GCWSA 2

21 Experiment: Multi-level Deblurring Ground truth image 2 pixel motion blur at 45 deg. Blurred image 15X15 synthetic image Pixel values, L = {1, 2, 3, 4, 5} Linear motion blur, length 45 o AWGN for 1 db SNR. J prior : otal-variation Noisy blurred image Joint Inversion and Labeling 21

22 Summary so far GCWSA: New approximation-based method for graph-cut solution of discrete linear inverse problems Based on separable approximation of data term Question: Can structure be exploited? Are there special classes of operators C which lead to submodular problems? No approximation is necessary in this case! Answer: Yes!... next 22

23 Variable flipping/re-labeling Submodularity condition (binary case): p,q (,) p,q (1,1) p,q (,1) p,q (1, ) Idea of variable re-labeling/flipping (Hammer 1984, Kolmogorov 27): p q q p 1 1 re-label q q 1 q 1 1 Non-submodular term Submodular term In which graphs can you get all submodular interactions through relabelling? p q p q p q Non-submodular Submodular r frustrated cycle r s 23

24 Flipping Schemes Lemma 1. Flipping scheme for a set of nodes represented by an acyclic graph to make all interactions submodular: d(r, p)=1 d(r, q)=2 d(r, s)=2 r p q s x 1 if the pair i, j is nonsubmodular w(i, j) if the pair i, j is submodular Flip nodes with an odd distance to a root node Non-submodular Submodular 24

25 Flipping Schemes heorem. A flipping scheme for a set of arbitrarily connected nodes exists iff there is an even number of non-submodular interactions in each cycle p q r Yes No Flipping Scheme: Flip any spanning tree of the graph as given in Lemma 1. d = 5 d = 4 d = 3 d = 2 d = 1 root Non-submodular Submodular d = 6 d = 1 25

26 Implications for discrete inverse problems J Boston University Slideshow 2 itle Goes Here data ( x) y Cx y y 2y 2 Cx x C Cx x 1 x 1 x 1 x 2 x 1 x 3 x 1 x 4 x 1 x N x 2 x 1 x 2 x 2 x 2 x 3 x 2 x 4 x 2 x N Structure of C C : 1. First off-diagonal acyclic interactions 2. Odd off-diagonals even ( flipable ) cycles 3. Even off-diagonals odd (frustrated) cycles x 3 x 1 x 3 x 2 x 3 x 3 x 3 x 4 x 3 x N x 4 x 1 x 4 x 2 x 4 x 3 x 4 x 4 x 4 x N x N x 1 x N x 2 x N x 3 x N x 4 x N x N 1. First off-diagonal Odd off-diagonals Even off-diagonals

27 Submodular Idea #2: Special class of systems: No odd cycles hese systems are guaranteed graph representable! 27 Cx C x Cx y y y Cx y x J 2 ) ( 2 2 data Odd-banded and non-negative C C Ground truth image Blurred image Graph Cut Reconstruction C C e.g. Cholesky Graph-cut methods for joint reconstruction and labeling

28 Dual Decomposition Inversion Approach DD idea: Break single hard problem into multiple easy to solve problems Slave problems are solved independently Master coordinates, enforces consistency via Lagragian dual E.g. Loopy graph solution via overlapping trees, message passing Master Problem Master problem coordinates the slave problems.... Slave 1 Slave 2 Slave 3 Slave M Easily optimizable slave problems 28

29 Idea #3: Efficient DD slaves for inverse problems Decomposition exploiting structure All odd off-diagonals (boxes) form a single, submodular subproblem Each even off-diagonal forms a separate tree subproblem Number of subproblems ~ number of bands For FIR convolution problem, number of subproblems limited by size of kernel (not size of image) x 1 x 1 x 1 x 2 x 1 x 3 x 1 x 4 x 1 x N x 2 x 1 x 2 x 2 x 2 x 3 x 2 x 4 x 2 x N x 3 x 1 x 3 x 2 x 3 x 3 x 3 x 4 x 3 x N x 4 x 1 x 4 x 2 x 4 x 3 x 4 x 4 x 4 x N x N x 1 x N x 2 x N x 3 x N x 4 x N x N 29

30 Efficient master huristics After many iterations some node labels obtain obvious consensus Strong Nodes : Nodes where slaves agree Fix strong nodes and remove them from problem Greatly reduces frustrated cycles After 1 iterations: Pixel Label 1 Label 2 p 9 1 q 95 5 r

31 Experiment: Dual Decomposition Binary image Blurred image Blurred + noisy DD + no MRF QPBO-I + no MRF DD + MRF QPBO-I + MRF DD + MRF Noise Free Case Noisy Case 31

32 Experiment: Dual Decomposition Performance 3x3 Convolution kernel Method Missed Pixels Unlabeled Pixels QPBO - %65 QPBO-P %21 % QPBO-I %13 % GCDD % % 5x5 Convolution kernel Method Missed Pixels Unlabeled Pixels QPBO - %95 QPBO-P %48 % QPBO-I %19 % GCDD %7 % 32

33 Summary Idea #1: GCWSA graph-cut algorithm for general inverse problems. Based on separable approximation of data term Idea #2: Specification of conditions and flipping scheme for obtaining submodularity for an inverse problem Idea #3: Identification of special banded inverse problem structure for graph representability Idea #4: Exploitation of special structure for efficient graph-cut based dual decomposition 33