Mesh segmentation. Florent Lafarge Inria Sophia Antipolis - Mediterranee

Transcription

1 Mesh segmentation Florent Lafarge Inria Sophia Antipolis - Mediterranee

2 Outline What is mesh segmentation? M = {V,E,F} is a mesh S is either V, E or F (usually F) A Segmentation is a set of sub-meshes induced by a partition of S into k disjoint subsets segmentation can also be called partitioning or clustering

3 Outline Formulating a mesh segmentation problem Usually specifies by two key elements: a function measuring the quality of a partition, eventually under a set of constraints a mechanism for finding an optimal partition.

4 Outline Contents Attributes and constraints Segmentation algorithms

5 Outline How to measure the quality of a segmentation?

6 Outline What are we looking for? Planar clusters?

7 Outline What are we looking for? Planar clusters? Smooth clusters?

8 Outline What are we looking for? Planar clusters? Smooth clusters? Round clusters?

9 Outline What are we looking for? Planar clusters? Smooth clusters? Round clusters? Small/large clusters?

10 Outline What are we looking for? Planar clusters? Smooth clusters? Round clusters? Small/large clusters? Small number of clusters?

11 Outline What are we looking for? Planar clusters? Smooth clusters? Round clusters? Small/large clusters? Small number of clusters? Smooth boundary?

12 Outline What are we looking for? Planar clusters? Smooth clusters? Round clusters? Small/large clusters? Small number of clusters? Smooth boundary? structural clusters?

13 Outline Attributes and constraints Attributes: criteria used to measure the quality of a partition with respect to the input mesh attributes are usually embedded into some metrics

14 Outline Attributes and constraints Attributes: criteria used to measure the quality of a partition with respect to the input mesh attributes are usually embedded into some metrics Constraints: cardinality, geometry or topology of the clusters must be preserved (hard) or favored (soft)

15 Outline Attributes and constraints example of problems without constraints with hard constraints with sogt constraints min P f(p,attributes) min P f(p,attributes) under g(p)=0 min P f(p,attributes)-α.g(p)

16 Outline Attributes Distance and Geodesic distance

17 Outline Attributes Distance and Geodesic distance Planarity, normal direction

18 Outline Attributes Distance and Geodesic distance Planarity, normal direction Smoothness, curvature

19 Outline Attributes Distance and Geodesic distance Planarity, normal direction Smoothness, curvature Distance to complex geometric primitives

20 Outline Attributes Distance and Geodesic distance Planarity, normal direction Smoothness, curvature Distance to complex geometric primitives Symmetry

21 Outline Attributes Distance and Geodesic distance Planarity, normal direction Smoothness, curvature Distance to complex geometric primitives Symmetry Medial Axis, Shape diameter

22 Outline Attributes Distance and Geodesic distance Planarity, normal direction Smoothness, curvature Distance to complex geometric primitives Symmetry Medial Axis, Shape diameter Texture

23 Outline Constraints Cardinality Not too small/large or a given number of clusters Overall balanced partition Geometry Size: area, diameter, radius Convexity, Roundness Boundary smoothness Topology Connectivity (single component) Disk topology

24 Outline Contents Attributes and constraints Segmentation algorithms

25 Outline Segmentation algorithms Very large variety of «mechanisms» in the literature Common with image processing/computer vision Focus on some of them: Thresholding Region growing Hierarchical partitioning Markov Random Fields

26 Outline Thresholding simple one parameter for a binary segmentation h(i) i

27 Outline Hysteresis thresholding idea: to keep the sites connected (favor large clusters) two parameters for a binary segmentation t l < t h scheme threshold the sites with t l consider as cluster only the set of connected sites all > t l with at least one site > t h

28 Outline Region growing see primitive-based surface reconstruction slides Mechanism while the mesh is not entirely segmented choose an unlabeled site (a facet) assign label k to its similar adjacent sites iterate on the new assigned sites.. when propagation of cluster k finished, update k=k+1

29 Outline Hierarchical partitioning start from one region representing the entire mesh divide into several regions when the criteria are not valid, and iterate on the new regions

30 Hierarchical partitioning

31 Hierarchical partitioning

32 Hierarchical partitioning

33 Hierarchical partitioning

34 Outline Markov Random Fields (MRF) set of random variables having a Markov property described by an undirected graph Let V be the set of nodes in the graph Card(V) = number of random variables in the MRF

35 Outline Markov property in 1D (Markov chain) X 0 X 1 X n X n+1 n usually corresponds to time

36 Outline Markov property in 2D or on a manifold in 3D (Markov field) P[ X k X {Xk} ]= P[ X k (X n(k) )] with n(k) neighbors of k X n X k1 X k3 X m Xk2 X k X k4 X j

37 Outline Markov property in 2D or on a manifold in 3D (Markov field) P[ X k X {Xk} ]= P[ X k (X n(k) )] with n(k) neighbors of k X n X k1 X k3 X m X k2 X k X k4 X j

38 Outline Notion of neighborhood N= {n(i) /i V } is a neighborhood system if (a) i n(i) (b) I n(j) j n(i) a MRF is always associated to a neighborhood system defining the dependency between graph nodes

39 Outline MRF as an energy Gibbs energy (Hammersley-Clifford theorem) Let X be a MRF so that for all x, P(X=x)>0, Then P(X) is a Gibbs distribution of the form P(X=x)=exp U(x) U is called a Gibbs energy Z

40 Outline Markov property Why is the markovian property important? graph with 1M nodes if each node is adjacent to every other nodes: 1M*(999,999)/2 edges ~ 500 G edges each random variable cannot be dependent to all the other ones complexity needs to be reduced by spatial considerations

41 Outline Markov Random Fields for images two common graphs nodes = pixel centers edges = adjacent pixels (4-connexity) nodes = pixel corners edges = pixel borders =

42 Outline Markov Random Fields for meshes Graph nodes = vertices & graph edges = edges Graph nodes = facets & graph edges = edges Graph nodes = edges & graph edges = facets

43 Outline Bayesian formulation Let y, the data (attributes) x, the label we want to model the probability of having x knowing y Bayes law Posterior probability Likelihood Prior probability

44 Standard assumptions conditional independence of the observation P(Y=y X=x) = P(y i x i ) i V X is an MRF

45 From probability to energy data term : local dependency hypothesis (l=x) regularization : soft constraints Data term = -log (likelihhood) when Bayesian Regularisation term = - log (pairwise interaction prior) when Bayesian

46 Optimal configuration We search for the label configuration x that maximizes P(X=x Y=y) x = arg max Pr(X=x Y=y) x = arg min U(x) x

47 exercise: binary segmentation or? Graph structure Graph nodes = facets Graph edges = common edges Attributes on facet: [0,1] (y) labels: {white, black} (l) Energy: with yi if Di ( li) 1 yi V i, j ( l, l i j 0 ) 1 l ' white ' i otherwise if l l otherwise i j

48 jji exercise: binary segmentation or? Graph structure Graph nodes = facets Graph edges = common edges Attributes on facet: [0,1] (y) labels: {white, black} (l) Energy: with yi if Di ( li) 1 yi V i, j ( l, l i j 0 ) 1 l ' white ' i otherwise if l l otherwise i j Q1: what is the optimal configuration l if β =0? What is its energy? Q2: what is the optimal configuration if β inf? Q3: what are the other possible optimal configurations in function of β?

49 exercise: binary segmentation or?

50 Finding the optimal configuration of labels Graph-cut based approches fast but restrictions on energy formulation Monte Carlo sampling slow but no restriction

51 Optimization by graph-cut Find a cut in the graph separating the nodes into different groups

52 Etiquetage binaire Min-cut formulation sink n-links t a cut s source Capacity of a cut (S,T) : c S, T c x, y x S y T

53 labels labels Etiquetage non-binaire Multi-label cut t cut cut L(p) y y s p x x

54 α -swap each variable taking label α or can change its label to α or all α -moves are iteratively performed till convergence

55 α -swap Requirements V(l i,l i )=0 V(l i,l j )=V(l j,l i ) 0

56 α-expansion each variable either keeps its old label or changes to α all α-moves are iteratively performed till convergence

57 α-expansion Requirements V(l i,l i )=0 V(l i,l j )=V(l j,l i ) 0 V(l i,l j ) + V(l j,l k ) V(l i,l k )

58 Optimisation : a expansion α-expansion: example

59 Optimisation : a expansion -expansion -expansion -expansion -expansion -expansion -expansion -expansion

60 Optimization by Monte Carlo sampling simulating a Markov chain configuration space on the X 0 X 1 X 2 X 3 For all x, x 1, x 2, x n,

61 Optimization by Monte Carlo sampling the transitions correspond to perturbations of the current configuration. They are proposed according to a proposal density Q (or «kernel»)

62 Optimization by Monte Carlo sampling the chain is built to be ergodic in order to ensure the convergence toward a stationary measure (specified via an energy)

63 Optimization by Monte Carlo sampling the chain is built to be ergodic in order to ensure the convergence toward a stationary measure (specified via an energy) reversibility The probability to be in the configuration x AND to move to the configuration y is equal to the probability to be in the configuration y AND to move to the configuration x (detailed balance condition)

64 Optimization by Monte Carlo sampling the chain is built to be ergodic in order to ensure the convergence toward a stationary measure (specified via an energy) reversibility aperiodicity Avoid to fall in cycles in the chain For all x, Q(x x) >0

65 Optimization by Monte Carlo sampling the chain is built to be ergodic in order to ensure the convergence toward a stationary measure (specified via an energy) reversibility aperiodicity irreductibility any configuration in can be reached under from any initial configuration in, in finite time

66 Optimization by Monte Carlo sampling each iteration is composed of 2 steps proposition of a new state decision of moving to this new state or staying in the current one

67 Markov Chain Monte Carlo: algorithm At the iteration t, if the current state is X t =x, simulate according to the proposal density configuration y close to the configuration x, a new simulate accept X t+1 =y if, and keep X t+1 =x otherwise.

68 Markov Chain Monte Carlo: algorithm At the iteration t, if the current state is X t =x, simulate according to the proposal density configuration y close to the configuration x, a new simulate accept X t+1 =y if, and keep X t+1 =x otherwise. The density h is specified by an energy U such that

69 MCMC as an optimization technique Introduction of a stochastic relaxation: The relaxation parameter (called temperature) tends to 0 as t tends to infinity allows the control of the configuration space exploration by imposing to the process to be more and more selective. 7

70 MCMC as an optimization technique At the iteration t, if the current state is X t =x, simulate according to the proposal density configuration y close to the configuration x, a new simulate accept X t+1 =y if, and keep X t+1 =x otherwise.

71 An image segmentation problem nb pixels configuration space: = {1,2,3,4,5,6} energy: Gaussian likelihood + Potts model

72 Example: mesh segmentation with principal curvature attributes & soft geometric constraints Multi-label energy model of the form with V, set of vertices of the input mesh E, set of edges in the mesh l i, the label of the vertex i among : planar (1), developable convex (2), developable concave (3) and non developable (4)

73 Data term with

74 Data term with

75 Soft constraints Label smoothness Edge preservation with

76 Optimization

77 Some results