Markov Random Fields in Image Segmentation

Transcription

1 Preented at SSIP 2011, Szeged, Hungary Markov Random Field in Image Segmentation Zoltan Kato Image Proceing & Computer Graphic Dept. Univerity of Szeged Hungary

2 Zoltan Kato: Markov Random Field in Image Segmentation 2 Overview Segmentation a pixel labeling Probabilitic bili approach Segmentation a MAP etimation Markov Random Field MRF Gibb ditribution & Energy function Claical l energy minimization i i Simulated Annealing Markov Chain Monte Carlo MCMC ampling Example MRF model & Demo Parameter etimation EM

3 Zoltan Kato: Markov Random Field in Image Segmentation 3 Segmentation a a Pixel Labelling Tak 1. Extract feature from the input image Each pixel in the image ha a feature vector For the whole image, we have r f = { f : S } 2. Define the et of label Λ Each pixel i aigned a label l For the whole image, we have = {, S} } Λ For an N M image, there are Λ NM poible labeling. Which h one i the right egmentation? ti f r

4 Zoltan Kato: Markov Random Field in Image Segmentation 4 Probabilitic Approach, MAP Define a probability meaure on the et of all poible labeling and elect the mot likely one. P f meaure the probability of a labelling, given the oberved feature f Our goal i to find an optimal labeling ˆ which maximize P f Thi i called the Maximum a Poteriori MAP etimate: MAP ˆ = arg max P f Ω

5 Zoltan Kato: Markov Random Field in Image Segmentation 5 Bayeian Framework likelihood By Baye Theorem, we have prior P f P P f = P f P P f P f i contant We need to define P and P f in our model We will ue Markov Random Field

6 Zoltan Kato: Markov Random Field in Image Segmentation 6 Why MRF Modelization? In real image, region are often homogenou; neighboring pixel uually have imilar propertie intenity, color, texture, Markov Random Field MRF i a probabilitic bili model which capture uch contextual contraint Well tudied, d trong theoretical ti background Allow MCMC ampling of the hidden underlying tructure t Simulated Annealing Fat and exact olution for certain type of model Graph cut [Kolmogorov]

7 Zoltan Kato: Markov Random Field in Image Segmentation 7 What i MRF? To give a formal definition for Markov Random Field, we need ome baic building block Obervation Field and hidden Labeling Field Pixel and their Neighbor Clique and Clique Potential Energy function Gibb Ditribution

8 Zoltan Kato: Markov Random Field in Image Segmentation 8 Definition Neighbor For each pixel, we can define ome urrounding pixel a it neighbor. Example : 1 t order neighbor and 2 nd order neighbor

9 Zoltan Kato: Markov Random Field in Image Segmentation 9 Definition MRF The labeling field X can be modeled a a Markov Random Field MRF if 1. For all Ω : P Χ = > 0 2. For every Sand Ω : P, r = P, r N N r r denote the neighbor of pixel

10 Zoltan Kato: Markov Random Field in Image Segmentation 10 Hammerley-Clifford Theorem The Hammerley-Clifford Theorem tate that a random field i a MRF if and only if P follow a Gibb ditribution. 1 1 P = exp U = exp Z Z c C V c where Z = exp U i a normalization contant Ω Thi theorem provide u an eay way of defining MRF model via clique potential.

11 Zoltan Kato: Markov Random Field in Image Segmentation 11 Definition Clique A ubet C S i called a clique if every pair of pixel in thi ubet are neighbor. A clique containing n pixel i called n th order clique, denoted by. C n The et of clique in an image i denoted by C = C 1 U C U... U 2 C k ingleton doubleton

12 Zoltan Kato: Markov Random Field in Image Segmentation 12 Definition Clique Potential For each clique c in the image, we can aign a value V c which h i called clique potential ti of c, where i the configuration of the labeling field The um of potential of all clique give u the energy of the configuration U U = V c = VC V,... i + C i j c C i C 1 i, j C 2

13 Zoltan Kato: Markov Random Field in Image Segmentation 13 Segmentation of graycale image: A imple MRF model Contruct a egmentation model where region are formed by patial cluter of pixel with imilar intenity: Model parameter MRF egmentation model + find MAP etimate ˆ Input image egmentation ˆ

14 Zoltan Kato: Markov Random Field in Image Segmentation 14 MRF egmentation model Pixel label or clae are repreented by Gauian ditribution: P f Clique potential: 1 f exp μ 2σ = 2 2πσ Singleton: proportional to the likelihood of feature given : logpf. Doubleton: favour imilar label at neighbouring pixel moothne prior V c 2 i, j = βδ, i j β = + β if 2 = i j + if i j A β increae, region become more homogenou

15 Zoltan Kato: Markov Random Field in Image Segmentation 15 Model parameter Doubleton potential β le dependent on the input can be fixed a priori Number of label Λ Problem dependent uually given by the uer or inferred from ome higher level knowledge Each label l λ Λ i repreented by a Gauian ditribution Nµ λ,σ λ : etimated from the input image

16 Zoltan Kato: Markov Random Field in Image Segmentation 16 Model parameter The cla tatitic mean and variance can be etimated via the empirical mean and variance: where S λ denote the et of pixel in the training et of cla λ a training et conit in a repreentative region elected by the uer

17 Zoltan Kato: Markov Random Field in Image Segmentation 17 Energy function U Now we can define the energy function of our MRF model: 2 f μ = log 2πσ + + βδ, r 2σ, r Recall: Hence ˆ MAP P f = exp U = exp Z Z c C V c = arg max P f = arg min U Ω Ω

18 Zoltan Kato: Markov Random Field in Image Segmentation 18 Optimization Problem reduced to the minimization i i of a non-convex energy function Many local minima Gradient decent? Work only if we have a good initial egmentation Simulated Annealing Alway work at leat in theory

19 Zoltan Kato: Markov Random Field in Image Segmentation 19 ICM ~Gradient decent [Beag86]

20 Zoltan Kato: Markov Random Field in Image Segmentation 20 ICM iterated conditional mode x 2 mean oberved x 3 x 1 x 4 x 5 Simulated Annealing: accept a move even if energy increae with certain probability ICM Global min Can get tuck in local minima! Slide adopted from C. Rother ICCV 09 tutorial:

21 Zoltan Kato: Markov Random Field in Image Segmentation 21 Simulated Annealing Metropoli

22 Zoltan Kato: Markov Random Field in Image Segmentation 22 Temperature Schedule

23 Zoltan Kato: Markov Random Field in Image Segmentation 23 Temperature Schedule Initial temperature: et it to a relatively low value ~4 fater execution mut be high enough to allow random jump at the beginning! Schedule: Stopping criteria: Fixed number of iteration Energy change i le than a threhol

24 Zoltan Kato: Markov Random Field in Image Segmentation 24 Demo Download from:

25 Zoltan Kato: Markov Random Field in Image Segmentation 25 Summary Deign your model carefully Optimization i jut a tool, do not expect a good egmentation from a wrong model What about other than graylevel feature? Extenion to color i relatively traightforward

26 Zoltan Kato: Markov Random Field in Image Segmentation 26 What color feature? RGB hitogram

27 Zoltan Kato: Markov Random Field in Image Segmentation 27 Extract Color Feature We adopt the CIE-L*u*v* color pace becaue it i perceptually uniform. Color difference can be meaured by Euclidean ditance of two color vector. We convert each pixel from RGB pace to CIE- L*u*v* pace We have 3 color feature image L * u * v *

28 Zoltan Kato: Markov Random Field in Image Segmentation Zoltan Kato: Markov Random Field in Image Segmentation 28 Color MRF egmentation model Pixel label or clae are repreented by three-variate Gauian ditribution: 2 1 exp T n u f u f f P π r r r r Σ Σ = Clique potential: Singleton: proportional to the likelihood of f t i l Pf feature given : logpf. Doubleton: favour imilar label at neighbouring pixel moothne prior p p + = = = j i j i j i c if if j i V β β βδ,, 2 A β increae, region become more homogenou + j i if β

29 Zoltan Kato: Markov Random Field in Image Segmentation 29 Summary Deign your model carefully Optimization i jut a tool, do not expect a good egmentation from a wrong model What about other than graylevel feature? Extenion to color i relatively traightforward Can we egment image without uer interaction? Ye, but you need to etimate model parameter automatically EM algorithm

30 Zoltan Kato: Markov Random Field in Image Segmentation 30 Incomplete data problem Supervied parameter etimation we are given a labelled data et to learn from e.g. omebody manually aigned label to pixel How to proceed without labelled data? Learningg from incomplete data Standard olution i an iterative procedure called Expectation-Maximizationp Aign label and etimate parameter imultaneouly Chicken-Egg problem

31 31 EM principle : The two tep E Step : For each pixel, ue parameter to compute probability ditribution Parameter : Ppixel/labelPlabel Weighted labeling : Plabel/pixel M Step : Update the etimate of parameter baed on weighted or oft labeling

32 Zoltan Kato: Markov Random Field in Image Segmentation 32 The baic idea of EM Each of the E and M tep i traightforward auming the other i olved Knowing the label of each pixel, we can etimate the parameter Similar to upervied learning hard v. oft labeling Knowing the parameter of the ditribution, we can aign a label to each pixel by Maximum Likelihood ie i.e. uing the ingleton energie only without pairwie interaction

33 Zoltan Kato: Markov Random Field in Image Segmentation 33 Parameter etimation via EM Baically, we will fit a mixture of Gauian to the image hitogram We know the number of label Λ number of mixture component At each pixel, the complete data include The oberved feature f Hidden pixel label l a vector of ize Λ pecifie the contribution of the pixel feature to each of the label i.e. a oft labeling

34 Zoltan Kato: Markov Random Field in Image Segmentation Zoltan Kato: Markov Random Field in Image Segmentation 34 Parameter etimation via EM E tep: recompute l i at each pixel : = = λ λ λ λ λ P P P P P i f f f l λ Λ λ λ P P f M tep: update Gauian parameter for each label λ: each label λ:,..., = = S S P P S P P f f f f λ λ μ λ λ λ S P S f λ

35 Zoltan Kato: Markov Random Field in Image Segmentation 35 Summary Deign your model carefully Optimization i jut a tool, do not expect a good egmentation from a wrong model What about other than graylevel feature Extenion to color i relatively Can we egment image without uer interaction? Ye, but you need to etimate t model parameter automatically EM algorithm Can we egment more complex image? Ye, but then you need a more complex MRF model

36 Zoltan Kato: Markov Random Field in Image Segmentation 36 Color Textured Segmentation egmentation egmentation

37 Zoltan Kato: Markov Random Field in Image Segmentation 37 Color & Motion Segmentation

38 Zoltan Kato: Markov Random Field in Image Segmentation 38 Summary Deign your model carefully Optimization i jut a tool, do not expect a good egmentation from a wrong model What about other than graylevel feature Extenion to color i relatively Can we egment image without uer interaction? Ye, but you need to etimate model parameter automatically y EM algorithm Can we egment more complex image? Ye, but then you need a more complex MRF model What if we do not know Λ? Fully automatic egmentation require Modeling of the parameter AND a more ophiticated ampling algorithm Reverible jump MCMC

39 Zoltan Kato: Markov Random Field in Image Segmentation 39 MRF+RJMCMC v. JSEG X 500 RJMCMC 17 min JSEG Y. Deng, B.S.Manjunath: PAMI 01: 1. color quantization: color are quantized to everal repreenting clae that can be ued to differentiate region in the image. 2. patial egmentation: A region growing method i then ued to egment the image. JSE EG 1.5 min

40 Zoltan Kato: Markov Random Field in Image Segmentation 40 Benchmark reult uing the Berkeley Segmentation Dataet JSEG RJMCMC

41 Zoltan Kato: Markov Random Field in Image Segmentation 41 Reference Viit Forthcoming book: