Segmentation with non-linear constraints on appearance, complexity, and geometry

Transcription

1 IPAM February 2013 Western Univesity Segmentation with non-linear constraints on appearance, complexity, and geometry Yuri Boykov Andrew Delong Lena Gorelick Hossam Isack Anton Osokin Frank Schmidt Olga Veksler

2 Overview Standard linear constraints on segments intensity log-likelihoods, volumetric ballooning, etc. 1. Basic non-linear regional constraints enforcing intensity distribution (KL, Bhattacharia, L 2 ) constraints on volume and shape 2. Complexity constraints (label costs) unsupervised and supervised image segmentation, compression geometric model fitting 3. Geometric constraints unsupervised and supervised image segmentation, compression geometric model fitting (lines, circles, planes, homographies, motion, )

3 Image segmentation Basics E(S) = = f,s f s p + p p B(S) S Pr(I Fg) Pr(I Bg) f p Pr(I = ln Pr(I p p fg) bg) I 3

4 Linear appearance of region S R ( S ) = f, S Examples of potential functions f Log-likelihoods f p = ln Pr( I p ) Chan-Vese Ballooning f 2 p = ( I p c ) f p = 1

5 Part 1 Basic non-linear regional functionals p S 1 2 ( vol( S ) vol 0 ) p S ln Pr( I p S ) hist( S ) hist0

6 Standard Segmentation Energy Target Resulting Appearance Fg Probability Distribution Bg Intensity 6

7 Minimize Distance to Target Appearance Model p S ln Pr(I Pr(I p p fg) bg) R(S) = R(S) = KL( S S - T T 2 L ) Non-linear harder to optimize regional term R(S) = Bha( S, T ) 7

8 E(S) = R(S) + non-linear regional term B(S) S Ω appearance models shape 8

9 Related Work Can be optimized with gradient descent first order approximation Ben Ayed et al. Image Processing 2008, Foulonneau et al., PAMI 2006 Foulonneau et al., IJCV 2009 We use higher-order approximation based on trust region approach 9

10 a general class of non-linear regional functionals R(S) = F( f,s, 1, f k,s )

11 Regional Functional Examples Volume Constraint R(S) = ( 1,S V ) 2 0 S = 1,S f i,s f(x) = 1 f i (x) for bin i 11

12 Regional Functional Examples Bin Count Constraint R(S) = k Σ( i= 1 f i,s V ) i 2 S = 1,S f i,s f(x) = 1 f i (x) for bin i 12

13 Regional Functional Examples Histogram Constraint R(S) = S - T 2 L R(S) k ( P S) V ) = Σ ( = i 1 i i 2 P (S) i = f i,s 1,S 13

14 Histogram Constraint Regional Functional Examples R(S) = KL( S T ) R(S) = k ΣP i=1 i (S)log P i (S) V i P (S) i = f i,s 1,S 14

15 Histogram Constraint Regional Functional Examples R(S) = Bha( S ), T R(S) = log k Σ i= 1 P (S) i V i P (S) i = f i,s 1,S 15

16 Shape Prior Volume Constraint is a very crude shape prior Can be generalized to constraints for a set of shape moments m pq 16

17 Shape Prior Volume Constraint is a very crude shape prior f (x,y) = pq x 1 2 y 02 3 m pq (S) = f, S pq f pq (x,y) = x p y q 17

18 Shape Prior using Shape Moments m pq m 00 = Volume (m,m01) 10 = Center Of Mass m m m m = Principal Orientation Aspect Ratio 18

19 Shape Prior using Shape moments Shape Prior Constraint R(S) = S T Dist(, ) R(S) = p+ q k (m pq (S) m pq (T)) 2 m pq (S) = f, S pq f pq (x,y) = x p y q 19

20 Optimization of Energies with Higher-order Regional Functionals E(S) = R(S) + B(S) 20

21 Gradient Descent (e.g. level sets) Gradient Descent First Order Taylor Approximation for R(S) First Order approximation for B(S) ( curvature flow ) Robust with tiny steps Slow Ben Ayed et al. CVPR 2010, Freedman et al. tpami 2004 Sensitive to initialization E(S) = R(S) + B(S) 21

22 Energy Specific vs. General Speedup via energy- specific methods Bhattacharyya Distance Volume Constraint Ben Ayed et al. CVPR 2010, Werner, CVPR2008 Woodford, ICCV2009 We propose trust region optimization algorithm for general high-order energies higher-order (non-linear) approximation 22

23 General Trust Region Approach An overview The goal is to optimize E(S) = R(S) + B(S) Trust region Trust Region Sub-Problem S 0 d min S S0 d ~ E (S) = U (S) + B(S) 0 E 0 First Order Taylor for R(S) Keep quadratic B(S) 23

24 General Trust Region Approach An overview The goal is to optimize E(S) = R(S) + B(S) d Trust Region Sub-Problem S 0 min S S0 d ~ E (S) = U 0 (S) + E 0 B(S) 24

25 Solving Trust Region Sub-Problem Constrained optimization ~ minimize E(S) = U s.t. S S (S) + B(S) d Unconstrained Lagrangian Formulation 0 0 minimize Lλ (S) = U0(S) + B(S) + λ S S0 Can be optimized globally using graph-cut or convex continuous formulation L 2 distance can be approximated with unary u terms [Boykov, Kolmogorov, Cremers, Delong, ECCV 06] 25

26 Approximating distance S S 0 dc,dc = C 0 2 dc s ds dist( C,C0 ) = 2 d0( p ) dp C [BKCD ECCV 2006]

27 Trust Region Standard (adaptive) Trust Region Control of step size d Lagrangian Formulation Control of the Lagrange multiplier λ λ λ 1 d 27

28 Spectrum of Solutions for different λ or d Newton step Gradient Descent Exact Line Search (ECCV12) 28

29 Volume Constraint for Vertebrae segmentation R(S) 0 V min V max S Log-Lik. + length Initializations Log-Lik. + length + volume Fast Trust Region 29

30 Appearance model with KL Divergence Constraint Init Fast Trust Region Exact Line Search Gradient Descent Appearance model is obtained from the ground truth 30

31 Appearance Model with Bhattacharyya Distance Constraint Fast Trust Region Exact Line Search Gradient Descent Appearance model is obtained from the ground truth 31

32 Shape prior with Tchebyshev moments for spine segmentation Second order Tchebyshev moments computed for the user scribble Log-Lik. + length + Shape Prior Fast Trust Region 32

33 Part 2 Complexity constraints on appearance

34 Segment appearance? sum of log-likelihoods (linear appearance) histograms mixture models now: allow sub-regions (n-labels) with constraints based on information theory (MDL complexity) based on geometry (anatomy, scene layout)

35 Natural Images: GMM or MRF? are pixels in this image i.i.d.? NO! 35

36 Natural Images: GMM or MRF? 36

37 Natural Images: GMM or MRF? 37

38 Natural Images: GMM or MRF? 38

39 Binary graph cuts [Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004] 39

40 Binary graph cuts [Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004] 40

41 Binary graph cuts [Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004] 41

42 A Spectrum of Complexity Objects within image can be as complex as image itself Where do we draw the line? Gaussian? GMM? MRF? object recognition?? 42

43 Single Model Per Class Label Pixels are identically distributed inside each segment 43

44 Multiple Models Per Class Label Now pixels are not identically distributed inside each segment 44

45 Multiple Models Per Class Label Our Energy Now pixels are not Supervised identically distributed inside Zhu each & segment Yuille! 45

46 Leclerc, PAMI 92 Zhu & Yuille. PAMI 96; Tu & Zhu. PAMI 02 Unsupervised clustering of pixels color similarity + boundary length + MDL regularizer α-expansion (graph cuts) can handle such energies with some optimality guarantees [IJCV 12,CVPR 10] 46

47 (unsupervised image segmentation) Fitting color models information theory (MDL) interpretation: = number of bits to compress image I losslessly Λ + + = L L L N q p q p p p I h L L w L p E ) ( ) ( ) ( ), ( L L δ δ ) Pr( ln p I p L each label L represents some distribution Pr(I L)

48 (unsupervised image segmentation) Fitting color models Label costs only Delong, Osokin, Isack, Boykov, IJCV 12 (UFL-approach)

49 (unsupervised image segmentation) Fitting color models Spatial smoothness only [Zabih & Kolmogorov, CVPR 04]

50 (unsupervised image segmentation) Fitting color models Spatial smoothness + label costs Zhu & Yuille, PAMI 1996 (gradient descent) Delong, Osokin, Isack, Boykov, IJCV 12 (a-expansion)

51 Fitting planes (homographies) Λ + + = L L L N q p q p p p h L L T w L p E ) ( ) ( ) ( ), ( L L δ Delong, Osokin, Isack, Boykov, IJCV 12 (a-expansion)

52 Fitting planes (homographies) Delong, Osokin, Isack, Boykov, IJCV 12 (a-expansion) Λ + + = L L L N q p q p p p h L L T w L p E ) ( ) ( ) ( ), ( L L δ

53 Back to interactive segmentation 53

54 EMMCVPR 2011 segmentation sub-labeling colour models 54

55 Main Idea Standard MRF: image-level MRF object GMM background GMM Two-level MRF: image-level MRF object MRF background MRF GMMs GMMs unknown number of labels in each group! 55

56 Our multi-label energy functional data costs smooth costs label costs E = p P D p ( f p ) + pq N w pq c( f p, f q ) + l L h l I( p : f p = l) Penalizes number of GMMs (labels) prefer fewer, simpler models MDL / information criterion regularize unsupervised aspect set of labels L GMMs object labels GMMs background labels Discontinuity cost c is higher between labels of different categories two categories of labels (respecting hard-constraints) 56

57 More Examples standard 1-level MRF 2-level MRF 57

58 More Examples standard 1-level MRF 2-level MRF 58

59 Beyond GMMs GMMs only GMMs + planes GMMs plane 59

60 Part 3 Geometric constraints on sub-segments

61 Towards biomedical image segmentation Sub-labels maybe known apriori - known parts of organs or cells - interactivity becomes optional Geometric constraints should be added - human anatomy (medical imaging) - or known scene layout (computer vision)

62 Illustrative Example We want to distinguish between these objects input what we want what mixed model gives mixed appearance model: 62

63 main ideas sub-labels with distinct appearance models (as earlier) basic geometric constraints between parts - inclusion/exclusion - expected distances and margins For some constraints globally optimal segmentation can be computed in polynomial time 63

64 Illustrative Example Model as two parts: light X contains dark Y Center object the only object that fits two-part model (no localization) 64

65 Our Energy Let over objects L and pixels P layer cake a-la Ishikawa 03 variables 65

66 Surface Regularization Terms Let over objects L and pixels P Standard regularization of each independent surface 66

67 Geometric Interaction Terms Let over objects L and pixels P Inter-surface interaction 67

68 So What Can We Do? Nestedness/inclusion of sub-segments ICCV 2009 (submodular, exact solution) Spring-like repulsion of surfaces, minimum distance ICCV 2009 (submodular, exact solution) Spring-like attraction of surfaces, Hausdorf distance ECCV 2012 (approximation) Extends Li, Wu, Chen & Sonka [PAMI 06] no pre-computed medial axes no topology constraints 68

69 Applications Medical Segmentation Lots of complex shapes with priors between boundaries Better domain-specific models Scene Layout Estimation Basically just regularize Hoiem-style data terms [4] 69

70 Application: Medical input our result 70

71 Application: Medical full body MRI two-part model 71

72 Application: Medical full body MRI 72

73 Conclusions linear functionals are very limited be careful with i.i.d. (histograms or mixture models) higher-order regional functionals use sub-segments regularized by complexity or sparcity (MDL, information theory) geometry (based on anatomy) literature: ICCV 09, EMMCVPR 11, ICCV 11,, IJCV12, ECCV 12