Non-negative Matrix Factorization for Multimodal Image Retrieval

Non-negative Matrix Factorization for Multimodal Image Retrieval Fabio A. González PhD Machine Learning 2015-II Universidad Nacional de Colombia F. González NMF for MM IR ML 2015-II 1 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 2 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 3 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 4 / 54

Content-Based Image Retrieval F. González NMF for MM IR ML 2015-II 5 / 54

Query by Visual Example F. González NMF for MM IR ML 2015-II 6 / 54

Low-level vs. High-level F. González NMF for MM IR ML 2015-II 7 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 8 / 54

Semantic Annotation using ML Source: Nuno Vasconcelos, UCSD, http://www.svcl.ucsd.edu/projects/qbse/ F. González NMF for MM IR ML 2015-II 9 / 54

An Example (1) F. González NMF for MM IR ML 2015-II 10 / 54

An Example (2) 1 Recall vs Precision Graph - Semantic models Histogram Intersection MetaFeatures Identity Kernel Sobel 0.8 0.6 Precision 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 Recall F. González NMF for MM IR ML 2015-II 11 / 54

Disadvantages Requires a training set with expert annotations, so it is a costly process F. González NMF for MM IR ML 2015-II 12 / 54

Disadvantages Requires a training set with expert annotations, so it is a costly process Does not scale for large semantic vocabularies F. González NMF for MM IR ML 2015-II 12 / 54

Disadvantages Requires a training set with expert annotations, so it is a costly process Does not scale for large semantic vocabularies The mapping from visual features to annotations may lose the visual richness F. González NMF for MM IR ML 2015-II 12 / 54

Scalability F. González NMF for MM IR ML 2015-II 13 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 14 / 54

Multimodality Text and images come naturally together in many documents Academic papers, books Newspapers, web pages Medical cases F. González NMF for MM IR ML 2015-II 15 / 54

Multimodal Retrieval Unstructured text associated to images may be used as semantic annotations Images and texts are complimentary information units Take advantage of interactions between both data modalities F. González NMF for MM IR ML 2015-II 16 / 54

Multimodal Retrieval Unstructured text associated to images may be used as semantic annotations Images and texts are complimentary information units Take advantage of interactions between both data modalities Problems: Text associated to images is not structured Unclear relationships between keywords and visual patterns Possible presence of noise in both data modalities F. González NMF for MM IR ML 2015-II 16 / 54

Multimodal Retrieval Unstructured text associated to images may be used as semantic annotations Images and texts are complimentary information units Take advantage of interactions between both data modalities Problems: Text associated to images is not structured Unclear relationships between keywords and visual patterns Possible presence of noise in both data modalities Retrieval scenarios: Cross-modal: find images based on a text query find text based on an image query (image annotation) Visual retrieval based on a visual query F. González NMF for MM IR ML 2015-II 16 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 17 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 18 / 54

The Netflix Competition Problem: prediction of user ratings for films (collaborative filtering) Y. Koren, R. Bell, and C. Volinsky, Matrix factorization techniques for recommender systems, Computer, vol. 42, no. 8, pp. 30-37, August 2009 F. González NMF for MM IR ML 2015-II 19 / 54

The Netflix Competition Problem: prediction of user ratings for films (collaborative filtering) Prize: $1 Million to the first algorithm able to improve Netflix own algorithm results by at least 10% Y. Koren, R. Bell, and C. Volinsky, Matrix factorization techniques for recommender systems, Computer, vol. 42, no. 8, pp. 30-37, August 2009 F. González NMF for MM IR ML 2015-II 19 / 54

The Netflix Competition Problem: prediction of user ratings for films (collaborative filtering) Prize: $1 Million to the first algorithm able to improve Netflix own algorithm results by at least 10% Won on Sept/21/2009 by BellKor s Pragmatic Chaos Y. Koren, R. Bell, and C. Volinsky, Matrix factorization techniques for recommender systems, Computer, vol. 42, no. 8, pp. 30-37, August 2009 F. González NMF for MM IR ML 2015-II 19 / 54

The Netflix Competition Problem: prediction of user ratings for films (collaborative filtering) Prize: $1 Million to the first algorithm able to improve Netflix own algorithm results by at least 10% Won on Sept/21/2009 by BellKor s Pragmatic Chaos Their approach used Matrix Factorization to build a latent-factor representation of users and movies Y. Koren, R. Bell, and C. Volinsky, Matrix factorization techniques for recommender systems, Computer, vol. 42, no. 8, pp. 30-37, August 2009 F. González NMF for MM IR ML 2015-II 19 / 54

The Latent-Factor Model movies factors movies r 11... r 1m q 11... q 1f p 11... p 1m users users..... =.......... r n1... r nm q n1... q nf p f 1... p fm factors R QP Q, P 0 n 5 10 5 m 1.7 10 4 {(i, j) r ij 0} 10 8 f 200 F. González NMF for MM IR ML 2015-II 20 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 21 / 54

Non-negative Matrix Factorization Problem: to find a factorization Optimization problem: X n m = W n r H r m min A,B X WH 2 s.t. W, H 0 is the Frobenius norm It is a non-convex optimization problem Solution alternatives: Gradient descendent methods Multiplicative updating rules F. González NMF for MM IR ML 2015-II 22 / 54

Multiplicative Rules Optimization problem: min W,H X WH 2 s.t. W, H 0 Incremental optimization: H aµ H aµ (W T X ) aµ (W T WH) aµ W ia W ia (XH T ) αµ (WHH T ) αµ F. González NMF for MM IR ML 2015-II 23 / 54

Divergence Optimization Optimization problem: min W,H D(X WH) = ij s.t. W, H 0 Multiplicative Rules: ( X ij log ) X ij (WH) ij X ij + (WH) ij i H aµ H W iax iµ /(WH) iµ aµ i W ia µ W ia W H aµx iµ /(WH) iµ ia µ H aµ F. González NMF for MM IR ML 2015-II 24 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 25 / 54

PCA and SVD Problem: X n m = W n r H r m Principal Component Analysis (PCA) X = UΣV W = UΣ 1 2, H = Σ 1 2 V PCA = SVD keeping the best Eigenvectors Columns of U are orthonormal There is not restriction on sign. D. D. Lee and H. S. Seung, Learning the parts of objects by non-negative matrix factorization, Nature, vol. 401, no. 6755, pp. 788-791, October 1999 F. González NMF for MM IR ML 2015-II 26 / 54

Latent Semantic Indexing (LSI) documents factors documents x 11... x 1m w 11... w 1r h 11... h 1m terms terms..... =.......... x n1... x nm w n1... w nr h r1... h rm X W H factors Documents are represented by the frequency of keywords (terms) Uses SVD to find the factorization Factors = semantic concepts Columns of W are orthonormal F. González NMF for MM IR ML 2015-II 27 / 54

NMF vs LSI W. Xu, X. Liu, and Y. Gong, Document clustering based on non-negative matrix factorization, in SIGIR 03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval. New York, NY, USA: ACM, 2003, pp. 267-273 F. González NMF for MM IR ML 2015-II 28 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 29 / 54

NMF for Multimodal Learning F. González NMF for MM IR ML 2015-II 30 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 31 / 54

Multimodal Representation F. González NMF for MM IR ML 2015-II 32 / 54

Bag-of-Features Image Representation F. González NMF for MM IR ML 2015-II 33 / 54

Semantic Space: Multimodal Latent Indexing (I) Objects are described by terms in a textual vocabulary and a visual vocabulary F. González NMF for MM IR ML 2015-II 34 / 54

Semantic Space: Multimodal Latent Indexing (I) Objects are described by terms in a textual vocabulary and a visual vocabulary Objects are mapped to a latent (semantic) space where objects are represented by a set of latent factors F. González NMF for MM IR ML 2015-II 34 / 54

Semantic Space: Multimodal Latent Indexing (I) Objects are described by terms in a textual vocabulary and a visual vocabulary Objects are mapped to a latent (semantic) space where objects are represented by a set of latent factors NMF is used to build the latent representation F. González NMF for MM IR ML 2015-II 34 / 54

Semantic Space: Multimodal Latent Indexing (I) Objects are described by terms in a textual vocabulary and a visual vocabulary Objects are mapped to a latent (semantic) space where objects are represented by a set of latent factors NMF is used to build the latent representation Three main tasks: Multimodal clustering Automatic image annotation Image retrieval F. González NMF for MM IR ML 2015-II 34 / 54

Semantic Space: Multimodal Latent Indexing (II) F. González NMF for MM IR ML 2015-II 35 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 36 / 54

Multimodal Clustering F. González NMF for MM IR ML 2015-II 37 / 54

Dual Multimodal Clustering F. González NMF for MM IR ML 2015-II 38 / 54

Semantic Space Visualization (I) F. González NMF for MM IR ML 2015-II 39 / 54

Semantic Space Visualization (II) F. González NMF for MM IR ML 2015-II 40 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 41 / 54

Image Annotation F. González NMF for MM IR ML 2015-II 42 / 54

Annotation Results F. González NMF for MM IR ML 2015-II 43 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 44 / 54

Image Retrieval Scenario: visual retrieval based on a visual query F. González NMF for MM IR ML 2015-II 45 / 54

Image Retrieval Scenario: visual retrieval based on a visual query Images are represented in a latent space using NMF F. González NMF for MM IR ML 2015-II 45 / 54

Image Retrieval Scenario: visual retrieval based on a visual query Images are represented in a latent space using NMF Some images in the database may not have text content associated F. González NMF for MM IR ML 2015-II 45 / 54

Image Retrieval Scenario: visual retrieval based on a visual query Images are represented in a latent space using NMF Some images in the database may not have text content associated The image query is projected the latent space F. González NMF for MM IR ML 2015-II 45 / 54

Image Retrieval Scenario: visual retrieval based on a visual query Images are represented in a latent space using NMF Some images in the database may not have text content associated The image query is projected the latent space Image are retrieved according to their latent space similarity F. González NMF for MM IR ML 2015-II 45 / 54

Retrieval Performance F. González NMF for MM IR ML 2015-II 46 / 54

Semantic Space Dimension F. González NMF for MM IR ML 2015-II 47 / 54

Outline 1 The Problem Content-based image retrieval Semantic image retrieval Multimodal image retrieval 2 Matrix Factorization The Netflix Prize Non-negative matrix factorization NMF vs SVD 3 NMF for Multimodal Learning Semantic space Multimodal clustering Image annotation Multimodal retrieval Application to histology images F. González NMF for MM IR ML 2015-II 48 / 54

Histology Image Retrieval F. González NMF for MM IR ML 2015-II 49 / 54

Semantic Back Projection F. González NMF for MM IR ML 2015-II 50 / 54

Performance F. González NMF for MM IR ML 2015-II 51 / 54

Performance F. González NMF for MM IR ML 2015-II 52 / 54

Retrieval Performance Example F. González NMF for MM IR ML 2015-II 53 / 54

Thanks! fagonzalezo@unal.edu.co http://dis.unal.edu.co/~fgonza F. González NMF for MM IR ML 2015-II 54 / 54