Latent Variable Models for Structured Prediction and Content-Based Retrieval

Transcription

1 Latent Variable Models for Structured Prediction and Content-Based Retrieval Ariadna Quattoni Universitat Politècnica de Catalunya Joint work with Borja Balle, Xavier Carreras, Adrià Recasens, Antonio Torralba

2 Scene Recognition airport mountain bar Gesture Recognition Hands Crossed Hands Crossed Hands Crossed Hands Opened Hands Opened Hands Opened

3 Scene Recognition mountain airport Many problems in barvision involve learning mappings from complex image spaces to semantic categories. Gesture Recognition Hands Crossed Hands Crossed Hands Crossed Hands Opened Hands Opened Hands Opened

4 Why Hidden Variables? Semantic Classes High dimensional Low dimensional Classic mixture model More general

5 Hands Crossed Hands Crossed Hands Crossed Hands Opened Hands Opened Hands Opened

6 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

7 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

8 Structured Prediction

9 Temporal Dependencies Part of Speech Tagging Gesture Recognition

10 Spatial Dependencies Image Annotation

11 Sequence Prediction Models Hidden variables summarize what is important about the past

12 Sequence Prediction Models Distributions over single strings. X is discrete set.

13 Hands Crossed Hands Opened... Discretize features

14 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

15 Weighted Automata Representation (WA) Operator Model Representation (OOM) Initial State Vector Function Parametrization Model Operators Describes the distribution as a dynamic process

16 Mapping from standard HMM to WA parametrization j i i j Forward-Backward Equations

17 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

18 Why Spectral Learning? Spectral Learning: Algebraic method for recovering model parameters from observable statistics. These methods exploit directly the markovianity of the process They are fast, simple and scale easily to large datasets Much faster than alternative approaches based on Expectation Minimization

19 Hankel Matrix Distribution generated by a WA with n states Sub-block defined by a basis

20 Duality between n-rank factorizations of Hankel and WAs =

21 Recovering Operators =

22 Spectral Method We can recover a parametrization for the distribution from (almost) any rank-n factorization of H. The spectral method uses the thin SVD factorization. Costs depends on number of prefixes and suffixes

23 Discrete Homogeneous HMM

24 Modeling paired sequences Conditional Joint

25 Experiments The task: Recognize actions in tennis (serve, hit, non-hit,...) S S H H NH H H NH NH NH NH S S S The experimental setting: Take sequences from 4 games and cut them in subsequences. Random partition sub-sequences into training and test. Evaluation metric: average F1 (geometric mean of precision and recall) S

26 The features Video Sequence HOG3D descriptors from player bounding boxes Codebook Hard Index of closest Codebook entry Soft The spectral method can also be used for the soft representation similarity to codebook entry

27 Results CRF Spectral Joint

28 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

29

30 Complex Image Space

31 Color important for outdoor images less important for indoor images

32 Latent classes can model variability

33 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

34 Global Ranking Model D Database elementary relevance functions global ranking function Q Training Queries

35 Global Ranking Model set of triplet constraints database item a is more relevant to q than item b Ranking Loss function

36 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

37 Ranking model with latent variables Mixture of specialized ranking functions One for each latent class Log-linear model Feature representation

38 Ranking model with latent variables Ranking Loss function Alternating optimization strategy

39 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

40 Parameter estimation constraints with non-zero loss subgradient with respect to The influence of query q in the update of relevance function g is weighted by the probability that q belongs to class g.

41 Parameter Estimation more negative = better ranking performance of class g for query q subgradient with respect to If class g predicts good rankings for constraints of query q, then the update will increase the probability that q belongs to g.

42 Outline Latent Variable Models for Structured Structure Prediction Problem Representing distributions using WA Spectral learning algorithm Examples Mixture Model for Content-Based Image Retrieval Global Ranking Model Mixture Ranking Model Learning a Ranking Function Experiments

43 SUN Dataset images, indoor and outdoor scenes. Images are annotated with object tags. Ground-truth relevance function derived from object annotations. 5 Random Partitions: database images train queries validation queries test queries novel-database

44 Ground-Truth Constraints We create ranking constraints for each train query: 1- Find top K database nearest neighbors according to ground-truth relevance function. 2- Sample L items from remaining items 3- Generate KL ranking triplets 320,000 total number of ranking triplets.

45 Model Comparisons Global SVM: Learns a single weighted combination of elementary relevance functions. Transductive SVM: For each test query, it learns a relevance function using ranking contraints from k nearest neighbors. Mixture: A mixture ranking model, the number of hidden states is chosen using validation queries. We report precision and recall curves for predicting the 100 most relevant items for each test query.

46 Results

47 Results

48 Latent Classes

49 Summary Hidden variables can be useful for a variety of problems involving complex data. Spectral learning methods are a good tool for inducing hidden structure. Future Directions Apply the spectral method to large-scale computer vision tasks. Spectral methods for unsupervised learning over structured data