Learning High-order Generative Texture Models

Transcription

1 1 / 17 Learning High-order Generative Texture Models Ralph Versteegen, Georgy Gimel farb and Pat Riddle Department of Computer Science, The University of Auckland

2 2 / 17 Outline Introduction and motivation Texture modelling with nested Markov-Gibbs random fields (MGRFs) Heterogeneous texture features Synthesis results Conclusion

3 3 / 17 Markov-Gibbs random fields (MGRF) MGRFs (a.k.a. MRFs, Markov networks, undirected graphical models): Max entropy probability distributions based on statistics of local features. Given H f (g obs ), produce P such that E P [H f (g)] H f (g obs ) Also applicable to texture classification and segmentation H f (g obs ) Training image g obs Statistics Model sample

4 4 / 17 Features and Interaction Structure An order k feature is an function f of k pixels together with the shape of its support (a clique). Each feature is duplicated across the image (spatial invariance): Offsets All instances of a feature Simple example: pairwise grey level difference (GLD): f (p 1, p 2 ) := p 1 p 2.

5 5 / 17 Limitations of Existing MGRF Models High-order interactions now widely recognised to be necessary for more expressive models. But too many choices hence manual selection. High-order generative models almost exclusively use linear filters. Fixed features are unsuitable for all textures (e.g. regular patterns: long range interactions).

6 5 / 17 Limitations of Existing MGRF Models High-order interactions now widely recognised to be necessary for more expressive models. But too many choices hence manual selection. High-order generative models almost exclusively use linear filters. Fixed features are unsuitable for all textures (e.g. regular patterns: long range interactions).

7 5 / 17 Limitations of Existing MGRF Models High-order interactions now widely recognised to be necessary for more expressive models. But too many choices hence manual selection. High-order generative models almost exclusively use linear filters. Fixed features are unsuitable for all textures (e.g. regular patterns: long range interactions).

8 6 / 17 Contributions Automatic selection of high-order texture features Learning heterogeneous MGRFs by nesting models Alternative approach to high order models: Generalisation of local binary patterns (LBPs), and their use for texture generation

9 6 / 17 Contributions Automatic selection of high-order texture features Learning heterogeneous MGRFs by nesting models Alternative approach to high order models: Generalisation of local binary patterns (LBPs), and their use for texture generation

10 6 / 17 Contributions Automatic selection of high-order texture features Learning heterogeneous MGRFs by nesting models Alternative approach to high order models: Generalisation of local binary patterns (LBPs), and their use for texture generation

11 6 / 17 Contributions Automatic selection of high-order texture features Learning heterogeneous MGRFs by nesting models Alternative approach to high order models: Generalisation of local binary patterns (LBPs), and their use for texture generation

12 7 / 17 MGRFs with a Set of Features S A MGRF is an exponential model over images (vectors of grey levels) g: P(g V) = 1 Z(V) exp(h S(g) V) Descriptive statistics: histograms H S (g) = [H f (g) : f S]. H f (g) is a histogram of values of feature function f across the image. A vector of parameters (feature weights) V = [V f : f S]. Normalisation term 1 Z(V)

13 7 / 17 MGRFs with a Set of Features S A MGRF is an exponential model over images (vectors of grey levels) g: P(g V) = 1 Z(V) exp(h S(g) V) Descriptive statistics: histograms H S (g) = [H f (g) : f S]. H f (g) is a histogram of values of feature function f across the image. A vector of parameters (feature weights) V = [V f : f S]. Normalisation term 1 Z(V)

14 7 / 17 MGRFs with a Set of Features S A MGRF is an exponential model over images (vectors of grey levels) g: P(g V) = 1 Z(V) exp(h S(g) V) Descriptive statistics: histograms H S (g) = [H f (g) : f S]. H f (g) is a histogram of values of feature function f across the image. A vector of parameters (feature weights) V = [V f : f S]. Normalisation term 1 Z(V)

15 7 / 17 MGRFs with a Set of Features S A MGRF is an exponential model over images (vectors of grey levels) g: P(g V) = 1 Z(V) exp(h S(g) V) Descriptive statistics: histograms H S (g) = [H f (g) : f S]. H f (g) is a histogram of values of feature function f across the image. A vector of parameters (feature weights) V = [V f : f S]. Normalisation term 1 Z(V)

16 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

17 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

18 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

19 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

20 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

21 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

22 8 / 17 Nesting MGRFs A nested model is a general exponential model 1 P i (g V i ) = P i 1 (g) Z(V i ) exp(h f (g) V i ) A base model, P i 1 model from the previous iteration Add a correction to the base model to meet new constraints E Pi [H f (g)] H f (g obs ) (g i : image generated from P i 1, g obs : training image) Select a feature f with largest disagreement (distance) in statistics dist(h f (g i ), H f (g obs )) to add to the model. Approximately learn new parameters to meet the new constraints (Changing base model unneeded.) Repeat until no more features exceed error threshold, then move to next type

23 9 / 17 Iterative Structure Learning: An Example Selecting and adding pairwise features one at a time: Training image P 0 sample Pairwise errors P 1 sample Pairwise errors P 2 sample Pairwise errors P 3 sample Pairwise errors P 4 sample

24 10 / 17 Nested MGRFs The base model may be any model, including ones that aren t MGRFs. Can build heterogeneous models. New feature functions may be built on top of previous ones, incrementally building up features to high orders.

25 11 / 17 Binary pattern (BP) features LBPs/BPs compare surrounding pixels p 1,..., p k to a central pixel p 0, result is a binary vector [p 1 p 0,... p k p 0 ], so there are 2 k histogram bins. Nearly invariant to changes in contrast and grey-level offset. Fast to compute.

26 11 / 17 Binary pattern (BP) features LBPs/BPs compare surrounding pixels p 1,..., p k to a central pixel p 0, result is a binary vector [p 1 p 0,... p k p 0 ], so there are 2 k histogram bins. Nearly invariant to changes in contrast and grey-level offset. Fast to compute.

27 12 / 17 Binary pattern (BP) features Star BP k s are a three parameter family of features: k, number of points, radius of the ring, angle of rotation. E.g. for k = 9 considered fixed set of 144 star BP 9 features Jagged star ( jagstar ) BPs have in addition: radius of second ring E.g. for k = 9 considered fixed set of 2638 jagstar BP 9 features

28 12 / 17 Binary pattern (BP) features Star BP k s are a three parameter family of features: k, number of points, radius of the ring, angle of rotation. E.g. for k = 9 considered fixed set of 144 star BP 9 features Jagged star ( jagstar ) BPs have in addition: radius of second ring E.g. for k = 9 considered fixed set of 2638 jagstar BP 9 features

29 13 / 17 Example: Nesting iterations First: adding pairwise GLD 2 features. Afterwards: adding jagstar BP 9 features.

30 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli Portilla & Simoncelli, A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients.

31 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

32 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

33 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

34 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

35 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

36 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

37 14 / 17 Synthesis examples Original Pairwise GLD 2 GLD 2 + Composed BP 5 GLD 2 + Jagstar BP 9 GLD 2 + Jagstar BP 13 Portilla & Simoncelli

38 15 / 17 Example: Selected features for D66 Selected jagstar BP 13 features:

39 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

40 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

41 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

42 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

43 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

44 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

45 16 / 17 Conclusion & Extensions High order MGRF texture models with learnt structures are promising and unexplored territory. BP features struggle to capture small details and edges. Potential/Planned Extensions: More diverse features, e.g. filters to capture edges Composed features, e.g. co-occurrences of BPs & hierarchical models Application to segmentation & classification; real world conditions Thanks for listening

46 17 / 17 Human synthesis evaluation 17 Human observers scored a set of synthesised images on a scale from 0 (= no resemblance) to 9 (=indistinguishable). Mean and standard deviation of scores were: Stochastic Near-regular Irregular Total GLD 2 3.7± ± ± ±1.8 GLD 3 4.2± ± ± ±1.9 BP 5 4.7± ± ± ±2.0 Conjoin BP 9 3.8± ± ± ±2.0 Star BP 9 4.0± ± ± ±1.9 Artifact of the procedure: using BP 5 features, was more likely to pick something rather than stopping.