Graph Cuts. Srikumar Ramalingam School of Computing University of Utah

Transcription

1 Graph Cuts Srikumar Ramalingam School o Computing University o Utah

2 Outline Introduction Pseudo-Boolean Functions Submodularity Max-low / Min-cut Algorithm Alpha-Expansion

3 Segmentation Problem [Boykov and Jolly 00, Rother et al. 004]

4 Stereo Reconstruction Choose the disparities rom the discrete set: (,,..., L)

5 Image Denoising Original Denoised image

6 Semantic Labeling (Building, ground, sky) [Hoiem, Eros, Hebert, IJCV, 007 ]

7 Image Labeling Problems

8 Labeling is highly structured Highly unlikely Image Courtesy: Lubor Ladicky

9 Labeling is highly structured Slide Courtesy: Lubor Ladicky

10 Image Labeling Problems Slide Courtesy: Lubor Ladicky

11 Outline Introduction Pseudo-Boolean Functions Submodularity Max-low / Min-cut Algorithm Alpha-Expansion

12 Pseudo Boolean Functions (PBF) Variables: x, x,..., 0, x n Negations: 0, xi x i Pseudo-Boolean Functions (PBF): :{0,} n» Maps a Boolean vector to a real number. R Has unique multi-linear representation:» For example: ( x, x, x, x ) 3x x x x x [Boros&Hammer 00]

13 Posiorms or Pseudo-Boolean unctions (PBF) Posiorms: Non-negative multi-linear polynomial except maybe the constant terms. ( x x x x, x, x3, x4) 3xx4 5 3( x ) x x x x 4 5 3x 3x x x 4 4 5xx 3 x4) 3xx4 5xx x4 3xx4 5xx 3 3( x 3 x Several posiorms exist or a given unction. Provides bounds or minimization, e.g [Boros&Hammer 00] 3

14 Set Functions are Pseudo Boolean Functions (PBF) Finite ground set V {,,..., n} Set unction (Input - subset o, output - real number) s : V V R - correspondence exists between and subset V S {,,3,4} o V. { x, x, x3 0, x4 } x (,,4), x,..., x n x x i i 0 0, i i S S

15 Set Functions are Pseudo Boolean Functions (PBF) Consider a PBF ( x, x, x3, x4) 3xx4 5xx3 Equivalent to a set unction s s ({,}) 3()(0) 5()(0) ({,3}) 3()(0) 5()() 7

16 Outline Introduction Pseudo-Boolean Functions Submodularity Max-low / Min-cut Algorithm Alpha-Expansion

17 Submodular set unctions (Union- Intersection) A set unction is submodular i and only i: V B A B A B A B A, ), ( ) ( ) ( ) ( R V : V B A,

18 Equivalent Deinitions Slide Courtesy: Krause, Jegelka

19 Questions Slide Courtesy: Krause, Jegelka

20 Submodularity Example Slide Courtesy: Krause, Jegelka

21 Submodularity Example Slide Courtesy: Krause, Jegelka

22 Submodularity Example Slide Courtesy: Krause, Jegelka

23 Submodularity Example Slide Courtesy: Krause, Jegelka

24 Set cover is submodular Slide Courtesy: Krause, Jegelka

25 Submodular set unctions (Union- Intersection) ( A) ( B) ( A B) ( A B), A, B V Let us consider a very simple case with only two variables x and x. V {,}, A {}, B {} (0,0) (0,) Using submodularity, we have: ( x ( x, x, x 0) ) ( x ( x ) 0) (,0) (0,) (,) (0,0) 0, x 0, x (,0) (, ) Main diagonal elements are smaller than o-diagonal ones. Blue is larger than red.

26 Quadratic Pseudo Boolean Functions (QPBF) Example o quadratic pseudo Boolean unctions ( x, x, x, x ) x 3x x x x x [Boros&Hammer 00]

27 Submodular Quadratic Pseudo Boolean Functions A QPBF is submodular i and only i all quadratic coeicients are non-positive. 3 x, x, x3) 5 x 3x xx 5 ( x x 3

28 Example or submodular QPBF ( x, x, x ) 5 x 3x 3x x x x V {,,3}, A {,}, B {,3} A B {,,3}, A B {} ( A) 5 3() 3()(0) 5()(0) 3 3 ( B) 5 0 3() 3(0)() 5()() 7 ( A B) 5 3() 3()() 5()() 5 ( A B) 5 0 3() 3(0)(0) 5()(0) ( A) ( B) ( A B) ( A B),(3 7 5 )

29 Outline Introduction Pseudo-Boolean Functions Submodularity Max-low / Min-cut Algorithm Alpha-Expansion

30 Max-low/Min-cut Image courtesy: Lubor Ladicky

31 Max-low/Min-cut

32 Max-low/Min-cut

33 Max-low/Min-cut

34 Network model or submodular QPBF A submodular QPBF can be associated with a network. There is - correspondence every edge in network and every term in. Let us denote source by s 0 and sink by t. x An edge that goes rom to x is denoted by x x. G v x x s x x x sx x t x t

35 Network model or submodular QPBF x x x x s x sx x x x xt t Given a QPBF we rewrite it using a posiorm representation using only three types o terms: x, x,, i x j i x i 3x x x x 4 3x x ( 4xx 4x 4x) 3x x 4( x ) x x 4 3 s 3x 3x x x 4 x 4 x 3x ( 3x 3 3) x x 4 3 3x 3( x ) x x 4 3 3sx x x 3xt 4 t 3

36 Network model or submodular QPBF There is a one-one correspondence between values o and s-t cut values o [Hammer 965] x G v. s 3 sx 4 x x 3 x t t x ( x C({ x 0, x, x }) ) 4 s-t mincut [Ford&Fulkerson 6, Goldberg&Tarzan86]

37 Network model or submodular QPBF There is a one-one correspondence between values o and s-t cut values o [Hammer 965]. s 3 4 G v x 3 x Thus we can compute the minimum o algorithm on the associated G v. t ( x C({ x, x, s},{ x using maxlow/mincut 0), t}) s-t mincut [Ford&Fulkerson 6, Goldberg&Tarzan86]

38 Network model or non-submodular QPBF A non-submodular QBPF can be associated with a network as ollows: 3x x 4x x x G v s x 3x 3sx 5x 5sx 4( x 4x x ) x t There is no polynomial-time algorithm or s-t mincut on a network with negative edge capacities. A submodular QBPF can always be associated with a network with non-negative edge capacities.

39 Minimizing Quadratic Pseudo Boolean Functions I QPBF is submodular, use maxlow algo.. [Ford&Fulkerson 6, Goldberg&Tarzan86] I QPBF is non-submodular, Belie propagation or other message passing algorithms. [Boros&Hammer 00]

40 Multi-label Problems Choose the disparities rom the discrete set: (,,..., L)

41 Multi-label Problems Exact Methods: Transorm the given multi-label problems to Boolean problems and solve them using maxlow/mincut algorithms or QPBO techniques. [Not covered in this course!] Approximate Methods: Develop iterative move-making algorithms where each move corresponds to a Boolean problem.

42 Outline Introduction Pseudo-Boolean Functions Submodularity Max-low / Min-cut Algorithm Alpha-Expansion

43 Boolean Energy Function Variables x, x,..., 0,. x n j - cost o assigning x i j {0, }. x i lm x x i j x i l x j m. - cost o jointly assigning and Energy unction: E( x j j j j ij i j, x) x x x x xx x x j0 j0 i0 j0

44 Multi-label Energy Function Variablesy, y,..., y 0,,...,. m L l yi y i l 0 otherwise. l yi - cost or assigning a single variable y i l. lm y y i j - cost o jointly assigning and y i l y j m. Energy unction: L L L L j y y y y y y y y j j i j j j j j ij i E( y, y ) mm/dd/yy 44

45 Energy Move Making Algorithms Solution Space [Image courtesy: Pushmeet Kohli, Phil Torr]

46 Energy Move Making Algorithms Current Solution Search Neighbourhood Optimal Move Solution Space [Image courtesy: Pushmeet Kohli, Phil Torr]

47 Expansion building [Boykov et al. 00]

48 Expansion Let yi and y j be two adjacent variables whose labels are not. retain y i l a y j retain lb In the move space, we compute i the two variables should retain the same labels or move to label. [Boykov et al. 00]

49 Expansion In the move space, we use two Boolean variables iand jto denote and y respectively. The encoding is shown below: y i y y i i ll a i j x i 0 x y x i Submodularity condition states that the sum o main diagonal elements is less than or equal to the sum o elements in the odiagonal: y j j ll j a x j j 0 x x 00 x i x j 0 x i x j 0 x i x j x i x j = l y a i i l y b lb y y j j a l y y i y i y j j [Boykov et al. 00]

50 Submodularity condition states that the sum o main diagonal elements is less than the sum o elements in the o-diagonal:

51 Expansion [Image courtesy: Lubor Ladicky] [Boykov et al. 00]

52 Swap The variables having the labels and can swap their labels or retain their previous states. y i y j retain retain [Boykov et al. 00] mm/dd/yy 5

53 Swap In the move space, we use two Boolean variables and to denote and respectively. The encoding is shown below: y x 0 y x 0 y i i y i y x j i i 0 Submodularity condition states that the sum o main diagonal elements is less than or equal to the sum o elements in the o-diagonal: 00 x i x j 0 x i x j 0 x i x j x i x j = mm/dd/yy 53 y j j y i y j y i y j y i y y i y j j x j j x i x [Boykov et al. 00] j

54 Swap Submodularity condition states that the sum o main diagonal elements is less than or equal to the sum o elements in the o-diagonal: 00 x i x j 0 x i x j 0 x i x j x i x j = y i y y i y Semi-metric condition: j j l y i y y i y a l l y y a a l l y y, l a b b j j L, 0, l l y y b a 0 [Boykov et al. 00] mm/dd/yy 54

55 Foreground / Background Estimation using Graph Cuts [Graph cuts slides adapted rom Lubor Ladicky] Rother et al. SIGGRAPH04

56 Foreground / Background Estimation Data term Smoothness term Neighborhood terms Data term Smoothness term Estimated using FG / BG colour models Delta unction is when the condition is satisied, and 0 otherwise Parameters that are manually set or learned rom data where Intensity dependent smoothness

57 Foreground / Background Estimation Data term Smoothness term

58 Foreground / Background Estimation Data term Smoothness term How to solve this optimization problem?

59 Foreground / Background Estimation Data term Smoothness term How to solve this optimization problem? Transorm into min-cut / max-low problem Solve it using min-cut / max-low algorithm

60 Reerences What energy unctions can be minimized using graph cuts? Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, Tutorial on Energy unctions (Longer version o the slides shown in the class with more details) Ishikawa s tutorial on graph cuts

61 Thank You