Multimedia Information Retrieval

Transcription

1 Multimedia Information Retrieval Norbert Fuhr HS-IR 98

2 Chapter 1 Introduction document structures and attributes media types terminology 1

3 Document structures and attributes IR networks heterogeneity effectivness user friendlyn. content structure head title author chapter section section chapter logical structure layout structure IR in networks J. Doe document author = J. Doe crdate = ladate = external attributes

4 audio Media types t y image text is a linear medium... video x

5 Terminology monomedia object/document: object containing data of a single media multimedia object/document: object containing data of multiple media hypertext document: nonlinear text document (i.e. with links) hypermedia document: nonlinear multimedia document

6 Course structure: 1. introduction 2. views on media 3. multimedia indexing

7 Chapter 2 Views on media views on media objects FERMI multimedia data model 6

8 2.1 Views on media objects here: images physical view pixel matrix logical view perceptive view colour texture brightness symbolic view spatial view: spatial relations (depending on modelling space) structural view set of image objects structural relations between image objects (aggregation)

9

10

11

12 2.2 The FERMI Multimedia Document Model Document structure and IR impact of structure: of multimedia information: heterogeneity of multimedia data on semantic content: logical structure ˆ= discourse structure on corpus classic IR: document = atomic unit MMIR: retrieval of document components

13 2.2.2 Elements of the multimedia data model logical structure hierarchy of structural objects leaves = single-media data implements explicit organization of discourse other data model elements refer to logical structure attributes classical attributes(author,dates,...) index expressions navigational structure links

14 The logical structure logical structure ˆ= hierarchical aggregation of structural objects: LS = (OS; str ; seq ; TYPE ST ; tst ; type st ; TYPE M ; type m ) OS: finite set of document structural objects elements: os i str : aggregative relation between structural objects, defines hierarchical composition seq : defines a linear sequence on OS (corresponds to standard, linear order to access components TYPE ST : set of types of structural objects e.g. for books: TYPE ST = fdocument; Chapter; Section; Sub Section; Paragraph; Figureg types correspond to abstraction levels tst : relation on structural object types defining hierarchy of abstraction levels type st : total function assigning each structural object its structural type in TYPE ST TYPE M : set of media types, TYPE M = ftext;image;graphic;multimediag. type m : total function assigning to each structural object its media type in TYPE M

15 Attributes A = (OS;NAME A ;VALUE A ; name a ;domain a ;value a ;SM) where: OS: the set of structural objects in the document elements: os i NAME A : set of attributes names. VALUE A : set of all possible attribute values (union of all the domain languages of all attributes) name a : partial function associating to structural objects a nonempty set of attribute names domain a : total function defining the domain of any attribute name (i.e. all the expressions of its associated language) value a : partial function assigning to structural objects the value for a related attribute name (definition allows multi-valued attributes)

16 Content Attributes single-media models involve up to five types of views: the physical view the structural view the symbolic view the spatial view (only in image and graphic models) the perceptive view (only in image and graphic models)! standard attribute names (called Content Attributes) for views: physical structural symbolic spatial perceptive

17 Indexing model indexing: assign index expressions to document structural objects retrieval of multimedia documents: retrieve smallest units that fulfill the query! index expressions assigned to parent object have to imply index expressions of its component objects index objects: structural objects that are indexed (assigned a value of attribute symbolic) index model of a document base: OI : set of index objects oi i OI OS I =(OI;TYPE I ; ind ) TYPE I : set of index object types TYPE I TYPE ST ind : relation representing structural dependency between index objects: ind OI OI

18 Example of an indexing structure example of structure and index hierarchy types (index objects of type Chapter or Subsection only) Structure Types Symbolic Types Document Chapter Section Subsection Paragraph

19 parts of the structural and semantic views of a document Structural View Semantic View Document Os1 U1 U2 Chapter Os2 Os3 Osem2 Osem3 Section Subsection Os8 Os4 Os5 Os6 Os7 U4 Os9 U6 Os10 Os11 Os12 Osem8 Osem9 Osem10 Osem11 Osem12 Paragraph Os13 Os14 Os15 Os16 Os17 U3 U5 Os18 Os19 U7

20 Chapter 3 Multimedia Indexing audio images video 19

21 3.1 Audio Sound retrieval E. Wold et al.: Content-based classification, search and retrieval of audio. IEEE Multimedia 3(3), pp Levels of audio retrieval 1. exact match of sound samples 2. inexact match of sounds, irrespective of sample rate, quantization, compresssion, inexact match of acoustic features / perceptual properties of sound 4. content-based match (for speech, musical content) here: inexact match of acoustic features and perceptual properties

22 Acoustic features aspects of sound considered: loudness root-mean-square of audio signal (in decibels) pitch greatest common divisor of peaks in Fourier spectra brightness centroid of short-time Fourier magnitude spectra (higher frequency content of signal) bandwidth magnitude-weighted average of differences between spectral components and the centroid (variation of frequencies, e.g. sine wave vs. white noise) harmonicity deviation of the sound s spectrum from a harmonic spectrum (i.e. harmonic spectra vs. inharmonic spectra vs. noise) variation of aspects over time: 1. compute aspect values at certain time intervals 2. derive features from sequences: average value variance autocorrelation (feature values weighted by amplitude)

23 sound example

24 Property Mean Variance Autocorrelation Loudness Pitch Brightness Bandwidth

25 Indexing and retrieval Indexing of a sound: compute and store feature vector a (mean, variance and autocorrelation for loudness, pitch, brightness, bandwidth and harmonicity) Retrieval: 1. conditions w.r.t. feature values 2. similarity of sounds: weighted Euclidian distance mean: µ= 1 M M a j j=1 covariance R = 1 M M (a j µ)(a j µ) T j=1 distance D = q (a b) T R 1 (a b) M # sounds considered

26 Property-based training and classification training: based on set of training sounds for a property (e.g. scratchiness) compute property-specific mean and covariance importance of feature: mean divided by standard deviation classification compute distances to means of all classes, select class with minimum distance likelihood: D 2 L = exp 2

27 Example: classification of laughter sounds

28 Example: class model for laughter Feature Mean Variance Importance Duration Loudness: Mean Variance Autocorrelation e Brightness: Mean Variance Autocorrelation Bandwidth: Mean Variance e Autocorrelation Pitch: Mean Variance Autocorrelation importance = jmeanj / p variance

29 3.1.2 Speech retrieval 1. speech recognition! uncertain term identification 2. application of text retrieval methods on recognized terms! TREC speech retrieval track Music retrieval McNab etal: The New Zealand Digital Library MELody index. D-Lib Magazine, May melody transcription 2. approximate string matching

30 3.2 Images Introduction Semantic vs. syntactic indexing and retrieval syntactic image features: color texture contour semantic image features: objects (humans, animals, buildings, art works) topics (pollution, demonstration, political visit) most image indexing methods support syntactic features only

31 Aboutness vs. ofness ofness: objects shown in the image aboutness: topic which is illustrated by the image aboutness is very much user-dependent e.g. image showing water pollution

32 3.2.2 QBIC tool for querying image and video databases example images user-constructed sketches and drawings selected color and texture patterns camera and object motion

33 System overview main components: database population: 1. processing of images and videos to extract syntactical features: colors textures shape camera motion object motion 2. storing features in database database querying 1. user composes query graphically 2. generate features from from graphical query 3. search for database objects with similar features

34

35

36 Data model basic elements: still images/scenes contain objects video shots sets of contiguous frames contain motion objects still images: scene: image or video frame object part of a scene videos: 1. break into clips (shots) 2. generate representative frame for each slot, treated as still image 3. generate motion objects from shots

37 querying: on objects images with a red, round object on scenes images with 30 % red and 20 % blue on shots shots panning from left to right on combinations images with 30 % red containing a blue object

38 Feature Calculation color color models: RGB, HSV, YUV, MTM average coordinates in color space k element histogram (typically k = 64;256)

39 texture coarseness: scale of texture contrast: vividness of a pattern (function of variance of grey-level histogram) directionality: peakedness of distribution of gradient directions in image (favoured direction (e.g. grass) vs. isotropic (e.g. sand))

40 shape area # pixels set in binary image circularity perimeter 2 /area major axis orientation 1. compute 2nd order covariance matrix from boundary pixels 2. major axis orientation = direction of largest eigenvector eccentricity = (largest eigenvalue) / (smallest eigenvalue) algebraic moment invariances consider 18 features invariant to affine transformations compute first m central moments as eigenvalues of predefined matrices

41 sketch based on reduced resolution edge map: 1. convert color image to single band luminance 2. compute binary edge image 3. reduce edge image to thin reduced image

42 Sample queries average color queries search for images/objects with similar color computed as weighted Euclidian distance in color space histogram color queries search for images with specified color distribution based on 256-element histogram: Q query histogram D image histogram Z element difference histogram: Z = Q D A symmetric color similarity matrix a(i; j)=1 d(c i ;c j )=d max c k kth color in histogram d(c i ;c j ) MTM color distance d max maximum distance between any two colors similarity: jjrjj = Z T AZ

43

44

45

46 texture queries user selects texture from a sampler compute weighted Euclidian distance in 3D texture space (coarseness, contrast, directionality) object shape user draws shape shape features: area, circularity, eccentricity, majoraxis-direction, object moments, tangent angles around object perimeter compute weighted Euclidian distance, weights are inverse variances of features query by sketch user draws dominant lines and edges 1. reduce user sketch to for each db image, correlate sketch with user sketch, based on edge/no edge comparison 3. compute correlation scores

47

48

49 3.2.3 IRIS semantic indexing of images 1. image analysis color contour texture 2. object recognition (a) basic objects: clouds, snow, water, sky, forest, grass, sand, stone (b) high-level objects: forestscene, skyscene, mountainscene, landscapescene,...

50 Image Analysis Color IRIS subdivides color space into about 20 different colors 1. subdivide image into nonoverlapping tiles 2. compute color histogram for each tile 3. most frequent color =: color of tile 4. join tiles with similar colors and compute circumscribing rectangle 5. compute attributes of color rectangles: position size color color density (# tiles with color / # tiles in rectangle) color evidence

51 original image

52 color-based segmentation:... colour2 HOR=mid,VER=up,SIZ=XL,SHP=Rect,COL=BLUE, UL=0 1,LR=44 11,DEN= colour3 HOR=mid,VER=mid,SIZ=M,SHP=Rect,COL=BLUE, UL=15 10,LR=44 17,DEN= colour4 HOR=left,VER=mid,SIZ=XS,SHP=Quad,COL=BLUE, UL=1 11,LR=1 11,DEN=1 1 colour5 HOR=left,VER=mid,SIZ=XS,SHP=Rect,COL=BLUE, UL=3 11,LR=14 12,DEN=

53 Texture consider local distribution and variation of grey values 1. compute normalized co-occurrence matrix p for 4 directions: 0,90,45, for each of the four directions, compute the following features from C: angular second moment contrast (local variations) correlation (linear relationship between pixel values) variance (deviation from the average) entropy 3. for each of the five parameters, compute the average from the values for the 4 directions (! invariance against rotation)

54 4. feed average values into neural network hidden-layer hidden-layer output-layer input-layer constrast asm variance correlation entropy forest gras sand water stone sky clouds ice 5. NN yields texture for each tile 6. join tiles with identical textures and compute circumscribing rectangles 7. compute attributes of texture rectangles: position size texture texture density (# tiles with texture / # tiles in rectangle)

55 ... texture3 HOR=mid,VER=mid,SIZ=L,SHP=Rect,TEX=ice, UL=2 2,LR=10 3,DEN=11 18 texture4 HOR=left,VER=mid,SIZ=S,SHP=Path,TEX=clouds, UL=0 3,LR=3 3,DEN=4 4 texture5 HOR=left,VER=mid,SIZ=S,SHP=Quad,TEX=stone, UL=4 3,LR=5 4,DEN=3 4 texture6 HOR=mid,VER=mid,SIZ=S,SHP=Rect,TEX=clouds, UL=5 3,LR=8 4,DEN=5 8...

56

57 Contour basedongreylevelimage 1. gradient-based edge detection 2. determination of object contours 3. shape analysis: compute position of centroid size of region bound coordinates of region

58

59 Object Recognition 1. step from syntactical to semantical features: identification of primitive objects 2. derivation of higher-level semantical features identification of primitive objects basis: color, texture and contour features for each feature, consider corresponding region form graph describing topological relationships between feature regions: node = feature edge = topological relationship: overlaps, meets, contains overlaps T contains CL T CT meets CT T CL CL CT T CT

60 formulate graph grammar rules for detecting primitive objects Mountainlake Clouds Texture Segment Sky Lake Mountain Clouds Forest Color Segment Contour Segment Conditions of "Clouds" predicate((valcompeq(*self(2,"colorseg","col"),"blue") valcompeq(*self(2,"colorseg","col"),"white")) && valcompeq(*self(2,"colorseg","ver"),"up")); predicate(nrkind(*self(1,"contourseg"),"contains",*self(1,"colorseg")) && nrkind(*self(1,"contourseg"),"contains",*self(1,"textureseg")));

61 3.2.4 Photobook developed at MIT Media Lab goal: semantic retrieval of images based on semantics-preserving image compression types of descriptions: appearance (faces) shape texture

62 Appearance based on eigenimage representations Training: Building Eigenrepresentations 1. preprocessing of input images: normalize w.r.t. position, scale, orientation 2. computation of eigenvectors of normalized image covariance for training images (faces) subregions of training images (eyes, nose, mouth)

63 mean and first few eigenvectors:

64 Retrieval Γ: new image (region) 1. transform Γ into face space 2. retrieval based on similarity measure

65

66

67 Shape representation based on modelling of physical deformations finite element method! stiffness matrix! eigenvectors Retrieval compute amount of energy needed to align object

68

69

70 Texture representation based on Wold decomposition for regular stochastic processes in 2D ˆ= sum of three orthogonal components: 1. harmonic field 2. generalized-evanescent field 3. purely-indeterministic field retrieval 1. derive parameters of Wold decompositions 2. compute similarity of parameter vectors

71

72

73 3.3 Video QBIC Representation of video data 1. shot detection 2. creation of representative frame 3. identify moving structures/objects Shot detection set of frames grouped into shots because they depict same scene signify single camera operation contain distinct event/action are chosen as single indexable unit

74 representative frame generation representative frames treated as still images in database population in retrieval returned for as answer representing shot representative frame generation methods: random frame from a shot synthesized r-frames mosaicking all frames in a panning shot remove moving objects

75

76 layered representation different layers used for identifying significant objects in the scene algorithm divides a shot into a number of layers, each with its own 2D affine motion parameters region of support in each frame

77

78

79 3.4 Summary: media indexing and matching 1. exact match 2. inexact media match (irrespective of digitization parameters) 3. inexact media feature match 4. content-based match

80 Chapter 4 Conclusions Issues in MMIR syntactic (signal-based) vs. semantic (symbolic) indexing of MM objects dealing with document structure IR models for multimedia documents 79