Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at CVPR 2012 Erik Sudderth Brown University

Transcription

1 Applied Bayesian Nonparametrics 5. Spatial Models via Gaussian Processes, not MRFs Tutorial at CVPR 2012 Erik Sudderth Brown University NIPS 2008: E. Sudderth & M. Jordan, Shared Segmentation of Natural Scenes using Dependent Pitman-Yor Processes. CVPR 2012: S. Ghosh & E. Sudderth, Nonparametric Learning for Layered Segmentation of Natural Images.

2 Human Image Segmentation

3 BNP Image Segmentation Segmentation as Partitioning! How many regions does this image contain?! What are the sizes of these regions? Why Bayesian Nonparametrics?! Huge variability in segmentations across images! Want multiple interpretations, ranked by probability

4 BNP Image Segmentation Model!! Dependent Pitman-Yor processes!! Spatial coupling via Gaussian processes Inference!! Stochastic search & expectation propagation Learning!! Conditional covariance calibration Results!! Multiple segmentations of natural images

5 Feature Extraction! Partition image into ~1,000 superpixels! Compute texture and color features: Texton Histograms (VQ 13-channel filter bank) Hue-Saturation-Value (HSV) Color Histograms! Around 100 bins for each histogram

6 Pitman-Yor Mixture Model PY segment size prior k 1 π k = v k (1 v l ) l=1 v k Beta(1 a, b + ka) Assign features to segments z i Mult(π) Observed features (color & texture) x c i Mult(θ c z i ) x s i Mult(θ s z i ) π z 1 z 2 z 3 z 4 x 1 x 2 x 3 x 4 Color: Texture: Visual segment appearance model

7 Dependent DP&PY Mixtures Some dependent prior with DP/PY like marginals Assign features to segments z i Mult(π i ) Observed features (color & texture) x c i Mult(θ c z i ) x s i Mult(θ s z i ) π 1 π 2 π 3 π 4 z 1 z 2 z 3 z 4 x 1 x 2 x 3 x 4 Color: Texture: Kernel/logistic/probit stick-breaking process, order-based DDP,! Visual segment appearance model

8 Example: Logistic of Gaussians! Pass set of Gaussian processes through softmax to get probabilities of independent segment assignments Fernandez & Green, 2002 Woolrich & Behrens, 2006 Figueiredo et. al., 2005, 2007 Blei & Lafferty, 2006! Nonparametric analogs have similar properties

9 Discrete Markov Random Fields Ising and Potts Models Previous Applications! Interactive foreground segmentation! Supervised training for known categories!but learning is challenging, and little success at unsupervised segmentation. GrabCut: Rother, Kolmogorov, & Blake 2004 Verbeek & Triggs, 2007

10 Region Classification with Markov Field Aspect Models Verbeek & Triggs, CVPR 2007 Local: 74% MRF: 78%

11 10-State Potts Samples States sorted by size: largest in blue, smallest in red

12 1996 IEEE DSP Workshop natural images giant cluster number of edges on which states take same value edge strength very noisy Even within the phase transition region, samples lack the size distribution and spatial coherence of real image segments

13 Geman & Geman, Iterations 128 x128 grid 8 nearest neighbor edges K = 5 states Potts potentials: 10,000 Iterations

14 Product of Potts and DP? Orbanz & Buhmann 2006 Potts Potentials DP Bias:

15 Spatially Dependent Pitman-Yor! Cut random surfaces (samples from a GP) with thresholds (as in Level Set Methods) π! Assign each pixel to the first surface which exceeds threshold (as in Layered Models) z 1 z 2 z 3 z 4 x 1 x 2 Duan, Guindani, & Gelfand, Generalized Spatial DP, 2007 x 3 x 4

16 Spatially Dependent Pitman-Yor! Cut random surfaces (samples from a GP) with thresholds (as in Level Set Methods)! Assign each pixel to the first surface which exceeds threshold (as in Layered Models) Duan, Guindani, & Gelfand, Generalized Spatial DP, 2007

17 Spatially Dependent Pitman-Yor! Cut random surfaces (samples from a GP) with thresholds (as in Level Set Methods)! Assign each pixel to the first surface which exceeds threshold (as in Layered Models)! Retains Pitman-Yor marginals while jointly modeling rich spatial dependencies (as in Copula Models)

18 Stick-Breaking Revisited 0 1 Multinomial Sampler: Sequential Binary Sampler:

19 PY Gaussian Thresholds Normal CDF because Gaussian Sampler: Sequential Binary Sampler:

20 PY Gaussian Thresholds Gaussian Sampler: Sequential Binary Sampler:

21 Spatially Dependent Pitman-Yor Non-Markov Gaussian Processes: PY prior: Segment size Normal CDF Feature Assignments

22 Preservation of PY Marginals Why Ordered Layer Assignments? Stick Size Prior Random Thresholds

23 Samples from PY Spatial Prior Comparison: Potts Markov Random Field

24 Inference!! Stochastic search & expectation propagation Outline Model!! Dependent Pitman-Yor processes!! Spatial coupling via Gaussian processes Learning!! Conditional covariance calibration Results!! Multiple segmentations of natural images

25 Mean Field for Dependent PY Factorized Gaussian Posteriors K Sufficient Statistics Allows closed form update of via K

26 Mean Field for Dependent PY Updating Layered Partitions Evaluation of beta normalization constants: K Jointly optimize each layer s threshold and Gaussian assignment surface, fixing all other layers, via backtracking conjugate gradient with line search Reducing Local Optima Place factorized posterior on eigenfunctions of Gaussian process, not single features K

27 Robustness and Initialization Log-likelihood bounds versus iteration, for many random initializations of mean field variational inference on a single image.

28 Alternative: Inference by Search Consider hard assignments of superpixels to layers (partitions) Marginalize layer support functions via expectation propagation (EP): approximate but very accurate Integrate likelihood parameters analytically (conjugacy) No need for a finite, conservative model truncation!

29 Maximization Expectation EM Algorithm!! E-step: Marginalize latent variables (approximate)! M-step: Maximize likelihood bound given model parameters ME Algorithm!! M-step: Maximize likelihood given latent assignments! E-step: Marginalize random parameters (exact) Why Maximization-Expectation? Kurihara & Welling, 2009!! Parameter marginalization allows Bayesian model selection!! Hard assignments allow efficient algorithms, data structures!! Hard assignments consistent with clustering objectives!! No need for finite truncation of nonparametric models

30 Discrete Search Moves Stochastic proposals, accepted if and only if they improve our EP estimate of marginal likelihood:!! Merge: Combine a pair of regions into a single region!! Split: Break a single region into a pair of regions (for diversity, a few proposals)!! Shift: Sequentially move single superpixels to the most probable region!! Permute: Swap the position of two layers in the order Marginalization of continuous variables simplifies these moves!

31 Inferring Ordered Layers Order A: Front, Middle, Back Order B: Front, Middle, Back!! Which is preferred by a diagonal covariance?!! Which is preferred by a spatial covariance? Order B Order A

32 Inference Across Initializations Mean Field Variational EP Stochastic Search Best Worst Best Worst

33 BSDS: Spatial PY Inference Spatial PY (MF) Spatial PY (EP)

34 Inference!! Stochastic search & expectation propagation Outline Model!! Dependent Pitman-Yor processes!! Spatial coupling via Gaussian processes Learning!! Conditional covariance calibration Results!! Multiple segmentations of natural images

35 Covariance Kernels! Thresholds determine segment size: Pitman-Yor! Covariance determines segment shape: Roughly Independent Image Cues:!! Color and texture histograms within each region: Model generatively via multinomial likelihood (Dirichlet prior)! Pixel locations and intervening contour cues: Model conditionally via GP covariance function probability that features at locations are in the same segment Berkeley Pb (probability of boundary) detector

36 Learning from Human Segments!! Data unavailable to learn models of all the categories we re interested in: We want to discover new categories!! Use logistic regression, and basis expansion of image cues, to learn binary are we in the same segment predictors:!! Generative: Distance only!! Conditional: Distance, intervening contours,!

37 From Probability to Correlation There is an injective mapping between covariance and the probability that two superpixels are in the same segment.

38 Low-Rank Covariance Projection!! The pseudo-covariance constructed by considering each superpixel pair independently may not be positive definite!! Projected gradient method finds low rank (factor analysis), unit diagonal covariance close to target estimates

39 Prediction of Test Partitions Heuristic versus Learned Image Partition Probabilities Learned Probability versus Rand index measure of partition overlap

40 Comparing Spatial PY Models Image PY Learned PY Heuristic

41 Inference!! Stochastic search & expectation propagation Outline Model!! Dependent Pitman-Yor processes!! Spatial coupling via Gaussian processes Learning!! Conditional covariance calibration Results!! Multiple segmentations of natural images

42 Other Segmentation Methods FH Graph Mean Shift NCuts gpb+ucm Spatial PY

43 Quantitative Comparisons Berkeley Segmentation LabelMe Scenes!! On BSDS, similar or better than all methods except gpb! On LabelMe, performance of Spatial PY is better than gpb Room for Improvement:!! Implementation efficiency and search run-time!! Histogram likelihoods discard too much information!! Most probable segmentation does not minimize Bayes risk

44 Multiple Spatial PY Modes Most Probable

45 Multiple Spatial PY Modes Most Probable

46 Spatial PY Segmentations

47 Conclusions Spatial Pitman-Yor Processes allow!!! efficient variational parsing of scenes into unknown numbers of segments!! empirically justified power law priors!! accurate learning of non-local spatial statistics of natural scenes!! promise in other application domains!

48 !but bravery is required Conclusions!! Conventional MCMC & variational learning prone to local optima, hard to scale to large datasets. But better methods on the way!!! Literature remains fairly technical. But growing number of tutorials!