Shifting from Naming to Describing: Semantic Attribute Models. Rogerio Feris, June 2014

Transcription

1 Shifting from Naming to Describing: Semantic Attribute Models Rogerio Feris, June 2014

2 Recap Large-Scale Semantic Modeling Feature Coding and Pooling Low-Level Feature Extraction Training Data Slide credit: Rogerio Feris

3 What if no training samples are available for the target class? Is this a practical setting? Slide credit: Rogerio Feris

4 Motivation ImageNet has 30 mushroom synsets, each with 1000 images. Slide credit: Christoph Lampert

5 Motivation In nature, there are 14,000 mushroom species. Zero-data: Many fine-grained visual categorization tasks may have classes with few or no training examples at all. Image: Slide adapted from Christoph Lampert

6 Motivation Suspect Search in Surveillance Videos [Feris et al, IBM] Zero-data: often no example images from suspects are available, only textual descriptions. Slide credit: Rogerio Feris

7 Motivation Prediction of concrete nouns from neural imaging data (mind reading) [Mark Palatucci et al, NIPS 2009] Noun Prediction Zero-Data: many nouns without corresponding neural image examples (costly label acquisition) Slide credit: Rogerio Feris

8 Motivation Similar problems in other fields: Large Vocabulary Speech Recognition Zero-Data: Infeasible to acquire training samples for each word (need sub-word modeling like phonemes) Recommendation Systems Zero-Data: Newly released apps without any user ratings (also known as cold-start problem ) [Schin et al, SIGIR 2002] Slide credit: Rogerio Feris

9 Semantic Attribute Models: Zero-Shot Learning for Visual Recognition [Lampert et al, CVPR 2009] [Farhadi et al, CVPR 2009] [Palatucci et al, NIPS 2009]

10 Attribute-based Classification Attributes: Semantic/nameable properties that are shared across classes Intuitive mid-level feature representation Slide adapted from Christoph Lampert

11 Attribute-based Classification [Lampert et al, CVPR 2009] Unseen categories Standard multi-class classification Unseen categories Semantic Attribute Classifiers Attribute-based classification Similar to Error-Correcting Output codes (ECOC [Dietterich & Bakiri, 1995]), but semantic codes are used instead Semantic Output Code Classifier (SOCC) [Palatucci et al, NIPS 2009] Slide credit: Rogerio Feris

12 Image-Attributes Prediction For each attribute, collect a set of positive and negative samples and train a classifier (e.g., using SVM or Neural networks) Example: Stripe Attribute Positive (Stripe) Negative (Non-Stripe) Binary Attribute Model Attributes transcend class boundaries Learning stripe attribute with images of zebras, clothing, Slide credit: Rogerio Feris

13 Image-Attributes Prediction Issue with Binary Attribute Models [Parikh and Grauman, ICCV 2011] Smiling??? Not smiling Natural??? Not natural

14 Image-Attributes Prediction Relative Attributes Replace binary model by a ranking function [Parikh and Grauman, ICCV 2011] i j i j Max-margin learning to rank formulation of Joachims 2002

15 Attribute-Class Associations Manual Specification of Class-Attribute Associations

16 Attribute-Class Associations Associations may be extracted automatically from other sources [Rohrbach et al, CVPR 2010]

17 Attributes as classes [Rohrbach et al, CVPR 2010] [Felix Yu et al, CVPR 2013] [Mensink et al, CVPR 2014] giant pandas are similar to grizzly and polar bears Attribute-based Direct similarity

18 Generalization: Label Embedding [Akata et al, CVPR 2013] Check talk by Florent Perronnin on Output embedding for large-scale visual recognition (LSVR CVPR 2014 tutorial)

19 Generalization: Label Embedding Label Embedding Framework Automatic Discovery of word associations Frome et al. "DeViSE: A Deep Visual-Semantic Embedding Model", NIPS 2013 Deep Learning Real-Value word vector representation Image Label Skip-gram model: Semantically related words are mapped to similar vector representations

20 Generalization: Label Embedding Label Embedding Framework Automatic Discovery of word associations [Frome et al, NIPS 2013] Zero-Shot Learning / Semantically close mistakes Language Model Source Code:

21 In addition to zero-shot classification, semantic attribute models have shown to be useful for many other tasks

22 Other Uses of Semantic Attributes Check the CVPR 2013 tutorial on Attributes: Slide credit: Rogerio Feris

23 Attribute-based Search Application: Smart Surveillance [Feris et al, IBM - WACV 2009, CVPR 2011, ICMR 2014]

24 Attribute-based People Search Slide credit: Rogerio Feris

25 Attribute-based People Search People Search in Surveillance Videos Traditional Approaches: Face Recognition ( Naming ) Face recognition is very challenging under lighting changes, pose variation, and lowresolution imagery (typical conditions in surveillance scenarios). Attribute-based People Search ( Describing ) Rather than relying on face recognition only, we provide a complementary people search framework based on fine-grained semantic attributes. Query Example: Show me all people with a beard and sunglasses, wearing a white hat and a patterned blue shirt, from all metro cameras in the downtown area, from 2pm to 4pm last Saturday". Slide credit: Rogerio Feris

26 Attribute-based People Search Suspect Description Form Slide credit: Rogerio Feris

27 Attribute-based People Search System Architecture Slide credit: Rogerio Feris

28 Attribute-based People Search Facial Attributes: bald, hair, color of hair, hat, color of hat, sunglasses, eyeglasses, absence of glasses, beard, mustache, absence of facial hair, skin tone (dark, medium,light), gender, Torso Attributes: clothing color, patterned, solid, Timestamp, Camera ID [Siddiquie et al, CVPR 2011] Slide credit: Rogerio Feris

29 Attribute-based People Search Attribute Ranking [Siddiquie, Feris and Davis, CVPR 2011] Learning to rank - confidence of individual attributes as features Pairwise attribute modeling Slide credit: Rogerio Feris

30 Structured Learning Formulation Improved performance over other ranking methods (RankSVM, RankBoost, DORM, TagProp) in three standard datasets (LFW, FaceTracer, PASCAL) See [Siddiquie, Feris and Davis, CVPR 2011] Slide credit: Rogerio Feris

31 Attribute-based People Search Top-1 Ranking Results [Feris et al, ICMR 2014] Slide credit: Rogerio Feris

32 Boston Bombing Event Show me all images of people matching the suspect description from time X to time Y from all cameras in area Z. Ability to spot a person with e.g., a white hat in a crowded scene 1071 detected faces from 50 high-res Boston images (all from Flickr) Suspect #1 found in 4 images in top 8 results Suspect #2 found in 3 images in top page Slide credit: Rogerio Feris

33 Extension to Vehicle Search [Feris et al, IEEE Trans on Multimedia, 2012] Show me all blue trucks larger than 7ft length traveling at high speed northbound last Saturday, from 2pm to 5pm. Slide credit: Rogerio Feris

34 Attribute-based Search Application: Product Search [Kovashka et al, CVPR 2012, ICCV 2013] [Yu & Grauman, CVPR 2014]

35 Whittle Search Slide credit: Kristen Grauman

36 Whittle Search Check Whittle Search demo at:

37 Resources

38 Resources

39 Resources Galaxy Morphological Attributes Data available at: 304,122 Galaxy Images 58,719,719 Annotations 83,943 volunteers 11 tasks / 38 answers (fine morphological attributes) Slide credit: Rogerio Feris

40 Resources 5 Terabytes of annotated data Data will be made publicly available soon! Slide credit: Rogerio Feris

41 Parts and Attributes Workshop (ECCV 2010) (ECCV 2012) (ECCV 2014) Check the Call for Extended Abstracts (Posters) Submission deadline: June 30th, 2014