Markov/Conditional Random Fields, Graph Cut, and applications in Computer Vision

Transcription

1 Markov/Conditional Random Fields, Graph Cut, and applications in Computer Vision Fuxin Li Slides and materials from Le Song, Tucker Hermans, Pawan Kumar, Carsten Rother, Peter Orchard, and others

2 Recap: Hammersley Clifford Any distribution that satisfy MRF independence (Markov blanket) can be represented as a Gibbs distribution,,, In square lattice: 1 exp Φ,, 1 exp Most images can be approximately viewed as square lattice!,,

3 Labeling in Computer Vision Assign each pixel a label (binary/multinomial) Binary Image Denoising Binary Image Segmentation

4 Standard Image Pairwise MRF Observed nodes (Image) Hidden nodes (Label on each pixel) Maximal cliques?

5 Image MRF Model,,,,, ; 1 exp, ; Note normalization constant is only dependent on, no dependency on and (summed over) Fixing in the joint defines a posterior ; MAP Inference = Energy minimization: max 1, exp, ;, ;,, ;, ; min, ;, ;,

6 Example: Binary Image Restoration Denoising Given (noisy) image I, recover intrinsic image X n Assume: Original image is piecewise smooth Goal: X should be smooth and close to I I I Adapted from A. Ishikawa

7 Example: Binary Image Restoration Find X that minimizes the energy E(X) I I

8 Example: Binary Image Restoration I I I

9 Example: Binary Image Restoration I I

10 Image MRF Model: Segmentation Suppose we want to do binary segmentation Observation: The (noisy) image(s) Prior? What is an appropriate prior for a segment? Likelihood? Why is a white object in the likelihood term? What is an appropriate likelihood for a white object?

11 Difficulty in Pairwise Term Boundaries are important in vision What if one wants to encode color difference between adjacent pixels, e.g., in the MRF? E.g. Penalize less when object boundary is at an image edge Sorry!

12 Conditional Random Field Instead of modelling the joint distribution of, focus only on modeling as a Gibbs distribution ; 1, exp ; Same MAP inference problem as MRF Difference? What do we lose? Why do we choose to lose that?, ;,

13 CRF for Image Segmentation Consider the interactive segmentation task Draw some strokes, then segment object using cues from these strokes

14 CRF for Image Segmentation Energy minimization ;, ;, Unary cost: 5 component GMM Color model Pixel color similar to foreground color model, low energy Pixel color similar to background color model, high energy Iteratively re estimate color model Pairwise cost: Pixel similarity on boundaries x, exp Only on boundaries!

15 We are set now!? User Initialisation Learn foreground color model MAP infer the foreground GrabCut Interactive Foreground Extraction 6

16 Iterated MAP Estimation? User Initialisation Learn foreground color model MAP infer the foreground GrabCut Interactive Foreground Extraction 6

17 Iterated MAP Estimation Result Energy after each Iteration GrabCut Interactive Foreground Extraction 7

18 Moderately straightforward examples GrabCut completes automatically GrabCut Interactive Foreground Extraction 10

19 Difficult Examples Camouflage & Low Contrast Fine structure No telepathy Initial Rectangle Initial Result GrabCut Interactive Foreground Extraction 11

20 Potts and Ising models Ising model used in the previous example Potts model: Pairwise term is, 1, Penalize connected nodes with different labels Ising model: When the label space is binary Besides, > Originated from solid state physics in studying ferromagnetism Both popular in computer vision Exact global optimal MAP solution with Ising model! There are extensions to approximately solve Potts model

21 Hardness of Map Estimation in MRF s

22 Energy minimization Find X that minimizes the Posterior Energy Function : E( X) ln p( X ) i I i ( i, j) V ( i, j) ( Xi, X j I) Data term (sensor noise) Smoothness term (MRF prior)

23 Graphs

24 Graphs and Minimum Cuts

25 Minimum Cut

26 Graph Cuts Basics: Simple 2D example Goal: divide the graph into two parts separating red and blue nodes s-t graph cut source S sink T A graph with two terminals S and T Cut cost is a sum of severed edge weights Minimum cost s-t cut can be found in polynomial time

27 Graph Formulation for Images Source: Simon Prince

28 Graph Cut = Labeling Source: Simon Prince

29 Graph Cut for Binary Image Restoration

30 Graph Cut for Binary Image Restoration

31 Mincut

32 Graph Cut Solution

33 Weights

34 Submodularity

35 Example Graphs

36 Example Graphs

37 3D Graphs

38 Minimum s-t cuts algorithms Augmenting paths [Ford & Fulkerson, 1962] O( VE Push-relabel [Goldberg-Tarjan, 1986] Boykov-Kolmogorov [Boykov and Kolmogorov, 2001] o No guarantee, relatively fast in practice Pseudoflow [Hochbaum 2008] o Fastest in practice 2 ) O(V 2 E )