Multimedia Data Management M

Transcription

1 ALMA MATER STUDIORUM - UNIVERSITÀ DI BOLOGNA Multimedia Data Management M Second cycle degree programme (LM) in Computer Engineering University of Bologna Semantic Multimedia Data Annotation Home page: Electronic version: 6.01.MultimediaDataAnnotation.pdf Electronic version: 6.01.MultimediaDataAnnotation-2p.pdf I. Bartolini

2 Outline Semantic gap in MM data retrieval Automatic annotation of MM data Imagination case study 2

3 Back to our main problem: MM data annotation Till now we have mostly focused on efficiency aspects i.e., How to efficiently execute a MM query? It is now time to also consider the effectiveness of the MM data retrieval process, which includes everything related to the user expectation! Effectiveness in term of: quality of result objects availability of simple but powerful tools, able to smooth the processes of query formulation/personalization result interpretation 3

4 The semantic gap problem Characterizing the object content by means of low level features (e.g., color, texture, and shape of an image) represents a completely automatic solution to MM data retrieval However low level feature are not always able to properly characterize the semantic content of objects e.g., two images should be considered similar even if their semantic content is completely different This is due to the semantic gap existing between the user subjective notion of similarity and the one according to which a low level featuresbased retrieval system evaluate two objects to be similar prevents to reach 100% precision results 4

5 Possible solution (Semi-)automatically provide a semantic characterization (e.g., by means of keywords or tags) for each object able to capture its content e.g., ([sky, cheetah] vs. [sky, eagle]) Combine visual features with tags by taking the best of the two approaches [LSD+06, LZL+07, DJL+08] [sky, cheetah] [sky, eagle] 5

6 Automatically infer semantics to MM objects Automatic objects annotation requires user intervention 1) Relevant feedback Exploiting user feedback to understand which are real relevant objects to the query will see how very soon 2) Learning The system is trained by means of a set of objects that are manually annotated by the user (training phase) Exploiting the training set, the system is able to predict labels for uncaptioned objects: the test object is compared to training objects; labels associated to the best objects are proposed for labeling (labeling & testing phases) DB images I. Bartolini user? sky, rock, ground, desert, Monument Valley 6

7 Learning approaches Classification MM object annotation as a problem of object classification (e.g., using Bayesian networks, SVM, K-means clustering etc.) in which each label is treated as a distinct class Graph-based Given the feature vector of a new object I, which is the probability that I belongs to class w? MM object annotation as a problem of graph exploration!a graph is a representation of a set of objects connected by links. The interconnected objects are represented by vertices/nodes, and the links that connect vertices are called edges Nodes represents images and its attributes (e.g., corresponding regions, associated labels, etc.) Edges are node-attribute relations Starting the navigation from the node representing the new image I, which is the probability to cross image node w? 7

8 Peculiarities on semantic annotation (1) Automatically infer semantics to MM objects is still an open challenge Image annotation techniques differ in: 1. What annotation means Enriching images with a set of tags/labels Providing a rich semantic description through the concepts of a full-fledged RDF ontology!an ontology is a formal framework for representing knowledge; it names and defines the types, properties, and interrelationships of the entities in a specific domain entities are conceptualizations of phenomena!resource Description Framework (RDF) is thew3c reference language for ontologies Based on triples (subject predicate object): the subject denotes the resource, and the predicate denotes traits or aspects of the resource and expresses a relationship between the subject and the object 2. What kind of tags/concepts are provided General-purpose system System tailored to discover only specific concepts/classes 8 8

9 Peculiarities on semantic annotation (2) 3. Annotation granularity Single keyword e.g., for images, at image level Multiple keywords e.g., for images, at image level or at region level 9

10 Image annotation problem How current commercial systems tackle the problem: Image search extensions of Google and Yahoo consider the original Web context, e.g.: file name title surrounding text to support keyword-based search Microsoft s Photo Gallery, Google Picasa, and Yahoo s Flickr rely on user-provided tags or labels Apple iphoto uses meta-data and user provided annotations Google similar images labs allows users to search for images using pictures rather than words (i.e., to find other images that look like the selected one) 10

11 Imagination case study [BC08a] Imagination: IMAGe (semi-)automatic annotation Images as set of regions Labels are tags which are associated at the image level Graph-based approach (à la Page Rank) 3-level of graph objects Images Regions with low level features (i.e., color and texture) Tags assigned to images plus K-NN links computed on region similarities Given a new image provide tags that are affine to the image and semantically correlated to each other 11

12 Intuitive example regions DB images new image I1 I2 I3 Iq tags deer, grass, bush bear, rock, grass, water, ground rock, bear, grass, water, ground?,,? 12

13 Mixed Media Graph (MMG) construction G MMG R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 I1 I2 I3 Iq T1 T2 T3 T4 T5 T6 T7 deer grass bush bear rock water ground DB images new image I. Bartolini deer, grass, bush bear, rock, grass, water, ground rock, bear, grass, water ground 13 13

14 Random Walk with Restart (RWR) [PYF+04] G MMG R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 I1 I2 I3 Iq restart! T1 T2 T3 T4 T5 T6 T7 deer grass bush bear rock water ground restart at the query node (with probability p) randomly walk to one link (with probability 1-p) For each tag node a relative frequency (i.e., the affinity) is computed approximating the steady state probability 14

15 Why tags correlation? MMG + RWR grass deer sheep horse cow ground new image predicted tags MMG + RWR heavily relies of NN edges involving the new image (i.e., low level features) If a region of the new image is highly similar to a region of G MMG, which however has some terms unrelated, this might easily lead to have such tags highly scored! uncorrelated tags, or even contradictory Tags correlation modeled as co-occurrence of pairs of tags within the DB images e.g., PSimRank algorithm (Fogaras et al. 2005) 15

16 Analyzing correlations of tags Link analysis on a sub-graph of G MMG to find highly-correlated tags bipartite graph G T second-order bipartite graph G T 2 G MMG R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 I1 I2 I3 G T T1 T2 T3 T4 T5 T6 T7 deer grass bush bear rock water ground 16

17 Intuitive example G T G 2 T (I2, I2) (bear, rock) (bear, grass) (bear, water) I2 T5 bear T4 (bear, tiger) (rock, rock) rock T3 grass (I1, I2) (rock, grass) (rock, water) I1 T2 water T1 (rock, tiger) (grass, grass) tiger (I1, I1) (grass, water) an edge between nodes (I i,i j ) and (T r,t s ) is added iff the two edges (I i,t r ) and (I j,t s ) equivalently, (I i,t s ) and (I j,t r )) are in G T edges E(u) node u (grass, tiger) (water, tiger) 17

18 PSimRank algorithm (Fogaras et al. 2005) A similarity score is computed for each tags node of G 2 T two nodes are similar if they are referenced by similar nodes two tags are similar if they are present in similar images two images are similar if they contain similar tags The process is independent from the query node off-line (I2, I2) (I1, I2) (bear, rock) (bear, grass) (bear, water) (bear, tiger) (rock, rock) (rock, grass) (rock, water) G 2 T (rock, tiger) Tags (grass, grass) Tags Similarity [0,1] (I1, I1) edges E(u) (grass, water) (grass, tiger) (water, tiger) node u 18

19 Putting it all together PT tags with highest steady state probability returned by MMG + RWR step are reduced considering tags correlation We model the problem as an instance of the Maximum Weight Clique Problem [BBP+99] An edge is added between two nodes if their correlation score exceeds a threshold c 19

20 Imagination user interface 20

21 Predicted tags 21

22 Semantic-based image retrieval (1) How to compare two images based on its annotation (i.e., its tags)? Exact text matching simple not flexible Query: flower flower petunia 22

23 Semantic-based image retrieval (2) Is it possible to perform better? Yes! Exploiting the semantic relations of keywords belonging to a preexistent taxonomy or ontology (e.g., WordNet [Mil95]) and applying a fuzzy text matching still simple more flexible more accurate Seed Plant Query: flower Flowering Plant Flower flower petunia Petunia Orchid Poppy Petunia is a flower! 23

24 Semantic relations Among possible semantic relations hyponymy/hypernymy (also indicated with subset/superset, or the ISA relation) Define hierarchy mammal A bear is a mammal bear feline canine brown bear black bear cat dog fox 24

25 Semantic similarity Problem: Quantify the similarity between two terms of the hierarchy (e.g., brown bear and feline) mammal bear feline canine brown bear black bear cat dog fox level(com-father) level(t 2 ) common-father t 3 t 2 Sim (t 1,t 2 ) = 2 * level(common-father) level(t 1 ) + level(t2) level(t 1 ) t 1 The similarity between a term of the hierarchy and terms belonging its sub-tree is equal to one (sim=1) 25

26 Semantic relaxation What happens if I am looking for bear images but neither bear images nor specialized term of bear images are present in the DB? To avoid empty semantic result, semantic relaxation is defined A weight for each level of the hierarchy (starting from the query term) represents the percentage of semantic relaxation the user is willing to accept W 0 =100 ROOT W 6 =68 animal weights W 7 =55 mammal W 8 =42 carnivore bear 26

27 Example Keyword: Bear Relaxation: 42% Carnivore Liv=7 Sim=0,933 Bear Liv=8 Feline Liv=8 Sim=0,875 Canine Liv=8 Sim=0,875 Brown bear Liv=9 Sim=1 Black bear Liv=9 Sim=1 Cat Liv=9 Sim=0,824 Dog Liv=9 Sim=0,824 2 *

28 Combining visual features with tags Content-based image retrieval (query by example (QBE) paradigm) + Semantic-based image retrieval (e.g., query by keyword paradigm) I am looking for flower images? flower 28

29 Practical example (1) content keyword: Flower semantic Flowering Plant Seed Plant Sem Sim=1 Sem Sim=1 Sem Sim=1 Flower Petunia Orchid Poppy Cont Sim=0,628 Cont Sim=0,486 Sem Sim=1 Cont Sim=0,431 Cont Sim=0,392 Sem Sim=1 Sem Sim=1 Sem Sim=1 Cont Sim=0,347 Cont Sim=0,339 Sem Sim=1 Sem Sim=1 Sem Sim=1 I. Bartolini 29

30 Practical example (2) Keyword: Flower livello 3 peso=90 Vascular Plant Relaxation=92% livello 4 peso=83 Seed Plant Ligneous Plant livello 5 peso=68 Flowering Plant Bush livello 6 peso=55 Flower Rose livello 7 peso=42 Petunia Orchid Poppy Sem Sim=1 Sem Sim( Flower, Rose )= Sem Sim=0,5 2*livello( Vascular Plant ) livello( Flower )+livello( Rose ) 30

31 Integration policies 1) Semantic similarity 2) Content similarity ,5 0,628 0, (Content similarity = null) 0,5 0,431 0,392 (Semantic similarity = null) Immagine Query 7 S:1 C:0,628 8 S:1 C:0,431 9 S:1 C:null 10 S:1 C:null 11 S:1 C:null 12 S:1 C:null S:1 C:null S:1 C:null S:0,5 C:null S:0,5 C:null S:null C:0,486 S:null C:0,392 31

32 petunia query 32

33 Visual result for petunia query 33