A System for Content-based Indexing and Retrieval in Multimedia Databases

Size: px

Start display at page:

Download "A System for Content-based Indexing and Retrieval in Multimedia Databases"

Rebecca Taylor
5 years ago
Views:

1 by Serkan KIRANYAZ. A System for Content-based Indexing and Retrieval in Multimedia Databases System Overview & Applications

2 MUVIS (v1.6) Overview FeX Modules AFeX Modules Fex & AFeX API Indexing Retrieval Still Images *.jp2 *.gif *.jpg *.bmp DbsEditor Database Creation *.wmf *.pcx *.tga *.eps *.pct *.png *.pgm *.yuv Database Editing Image and MM files - types Convertions Image and MM files Appending - Deleting Image Database An Image FeX- AFeX Management MBrowser Stored MM (Video-Audio) Encoding-Decoding- Rendering Hybrid Database Query Browsing Video Summarization Display A Frame AVDatabase AV Database Creation Real Time Video-Audio Capturing Encoding AV Database A Video- Audio Clip Recording Appending into Dbs.

3 MUVIS Multimedia Codecs MUVIS Audio MUVIS Video Sampling Freq. Channel File Formats Codecs Frame Rate FrameSize File Formats MP3 8 & Khz Mono MP3 H fps Any AVI AAC 12 & 16 KHz Stereo AAC MPEG-4 MP4 G KHz AVI YUV 4:2:0 G KHz MP4 RGB 24 PCM 32 KHz 44.1 KHz MUVIS Images JPEG JPEG 2K BMP TIFF PNG PCX GIF PCT TGA PCX EPS WMF PGM

4 Multimedia Databases 3 types of databases in MUVIS: Video Database vdbs : Containing video clips, keyframes and associated feature information Image Database idbs : Containing images and associated feature information Hybrid Database hdbs : Containing video clips, keyframes, images and associated feature information

5 Applications of MUVIS AVDatabase Real-time video indexing. Main audio/video database creator. Create and Append modes. Real time audio/video capturing, encoding, recording and display

6 Capturing: Applications of MUVIS.. AVDatabase Various picture formats, i.e. YUV 4:2:0, RGB24, YUY2, UYVY. Various frame rates, i.e. 1fps to 25fps Various sizes, i.e. QSIF, QCIF, SIF, CIF, etc. Video codecs: built in last generation codecs, i.e. MPEG-4, H263+. Audio/Video encoding parameters can be defined during recording time, i.e. VLBR to High bit-rates Capability of recording into interlaced (i.e. AVI, MP4) file formats. Support for several Audio codecs: PCM, G721, G723, MPEG 1, 2, 2.5 layer 3 (MP3), MPEG-2,4 AAC.

Applications of MUVIS.. DbsEditor Creating initial (dummy) audio/video or image MUVIS databases. Extracting new features or removing existing features of a database.

7 Applications of MUVIS.. DbsEditor Creating initial (dummy) audio/video or image MUVIS databases. Extracting new features or removing existing features of a database. Appending external audio/video clips and images to any MUVIS database. Removing existing audio/video clips or images from a MUVIS database. Previewing any audio/video clip or image. Enhanced Multimedia Conversions.

8 Applications of MUVIS.. MBrowser Main media retrieval and browser terminal Supports any kind of database and various image and video types Supports 5 levels of video display hierarchy: Single frame shot frames scene frames partial video (ROI) entire video Queried MM Clip PQ Browser Handle. 10th PSQ is currently active. PQ Knob First 12 Retrieval Results Prev Next

9 Applications of MUVIS.. MBrowser Supports user-friendly media browsing environment. Enables video summarization through shot (key) and scene frames. Region-of-interest (ROI) video handling and query options. Query based on merging of existing database features with user-defined weights. Supports cross querying between video clips, images and frames. Supports Fex & AFeX framework that aural/visual feature extraction modules that are used in run-time. Supports two Query Schemes: Progressive & Normal Queries.

10 Visual Feature Extraction in MUVIS Visual Features are extracted from key frames of video clips, and decoded images. Key frames of video clips are INTRA frames chosen by the video encoder, they might be stored in YUV format while recording. Generation of key frames (if not stored already) by decoding bit stream. Already implemented feature extraction modules are used Feature extraction module generates feature vectors Feature vectors are stored in their associated feature files.

11 Visual Feature Extraction (FeX) Framework in MUVIS Features represented by series of numbers, which correspond to feature vectors Normalization is required within a feature vector. Each feature vector should have unit length. This is required for multiple feature based retrieval. DbsEditor is the responsible application for the management of the feature extraction into any kind of multimedia database DbsEditor supports extraction of multiple features and multiple sub-features with different parameters from any single feature available.

12 FeX in MUVIS (cont.) MBrowser can be used to query any multimedia item from any MUVIS database, if there is at least on visual or aural feature extracted within the database. Features of video clips are extracted from the key-frames of the clips. Image features are extracted from 24bit RGB frame buffer of the decoded image. Feature extraction algorithms should be implemented as a Dynamic Link Library (DLL) exclusively with respect to the Feature Extraction Module Interface (FEX API), and stored in appropriate folder (i.e. C:\MUVIS\.)

13 FeX in MUVIS (cont.) Query item Feature Extraction Features Display Results Best M atches Sim ilarity M easurement End User Features M ultim edia Item s Features Features On-line Off-line Features M ultim edia Item s Feature Extraction M ultim edia Database Other Sources

14 FEX Module Interface (FeX API) Generic FeX API is defined in Fex_API.h file. The implemented FeX modules (DLLs) should include Fex_API.h file Five different API functions and necessary data structures are declared in Fex_API.h The name of the FeX DLL should be named as: Fex_[FourCC].dll (i.e Fex_TEXT.dll) A FeX DLL file should be stored in the same folder with the application (DBSEditor or MBrowser) or in C:\MUVIS\ folder

15 FEX Module Interface (cont.) DBSEditor Fex_*.DLL Fex_Bind Fex_Init Fex_Extract Fex_Exit Fex_API.h Fex_Bind() Fex_Init() Fex_Extract() Fex_GetDistance() Fex_Exit() Fex_GetDistance MBrowser

16 FEX_API.h : Structures FexParam Structure: The data members in the structure are set by the feature extraction module typedef struct{ char feat_name[255]; long feat_fourcc; unsigned int feat_param_no; long* feat_param_fourcc; double* feat_param_default; FrameType ftype; } FexParam; FrameParam Structure: Frame data is stored in the structure to be used for feature extraction typedef struct { unsigned char *buffer; int Xs, int Ys; } FrameParam;

17 FEX_API.h : Functions int Fex_Bind(FexParam*): Used for handshaking between a MUVIS application and a FeX module (DLL). Module introduces and binds itself to the application int Fex_Init(double*): Used to initialize the FeX module with given parameters double* Fex_Extract(FrameParam, int&): Used to extract the features of the frame given inside the FrameParam structure int Fex_Exit(FexParam*): Used to reset or terminate the FeX module double Fex_GetDistance(double*,double*,int): Used by only MBrowser to calculate the distance between two feature vectors

18 DBSEditor Feature Extraction Process 1. Looks for the FeX modules (DLLs) at startup, loads them and calls Fex_Bind of each module 2. When user enters feature extraction parameters, calls Fex_Init with the parameters to initialize the FeX module 3. For each frame in database, calls Fex_Extract with the FrameParam structure. Vector size should be returned as well as the feature vector. Feature vectors are stored 4. Calls Fex_Exit to reset the FeX module 5. To extract new features, step 2 to 4 can be repeated 6. While closing, calls Fex_Exit again

19 MBrowser Query & Feature Extraction Processes 1. Looks for the FeX modules (DLLs) at startup, loads and calls Fex_Bind of each module 2. When user queries a media, calls Fex_Init with the chosen parameters to initialize the FeX module 3. For each frame of queried media, calls Fex_Extract with the FrameParam structure. Vector size should be returned as well as the feature vector 4. Calls Fex_Exit to reset the FeX module 5. Calls Fex_GetDistance for each frame in database, with associated feature vectors and extracted feature vectors. Results are displayed accordingly 6. To query new media, step 2 to 5 can be repeated 7. While closing, calls Fex_Exit again

20 New Features on MUVIS v1.6 Audio-only clips are supported Now audio has the same priority as video. Audio based Multimedia Indexing & Retrieval is supported. New File Formats: MP3 & AAC. New Video Capturing Formats: YUY2, UYVY, RGB32 & RGB24.

21 New Features on MUVIS v1.6 (cont..) New Image Format: JPEG 2000 (*.jp2, *.j2c). A/V Direct Stream Access is supported KF storage of video clips in a MUVIS database is entirely optional. Image Conversion is supported into one of these formats: JPEG (with a compression quality factor) JPEG 2000 (with a compression rate) BMP PNG TIFF

22 A NOVEL MULTIMEDIA RETRIEVAL TECHNIQUE: PROGRESSIVE QUERY

23 Progressive Query (PQ) t 2t 3t 4t time Periodic Sub-Query Results Sub-Query Fusion Sub-Query Fusion Progressive Sub-Query Result

24 PQ Overview 1. Sub-Query Fusion Operation 2. Periodic Sub-Query Formation Atomic Sub-Query Sub-query rate estimation Fractional Sub-Query 3. Progressive Sub-Query Formation (PSQ)

25 PROGRESSIVE QUERY OVERVIEW Atomic Sub-Query No No items in ASQ >0 Implemented only at the beginnning of the first periodic sub-query Yes Calculate size of next Fractional Sub-Query Fusion Fractional Sub-Query No t > Tp-Tw Stop Yes Periodic Sub- Query(Tp)

26 Advantages of PQ (vs Normal Query) Reduced Retrieval Time (Faster Query Speed). Independent from Min. System Requirements. Memory & Speed

27 Advantages of PQ Query Accessibility Instantaneous Query Retrievals. Query Browsing. Query Stop Capability. The probability to retrieve better (more relevant) results than the final outcome due to the limited sub-set search within the database.

28 Audio-Based Multimedia Indexing and Retrieval Framework within MUVIS

29 Audio-Based Multimedia Indexing and Retrieval Framework for MUVIS A global framework implementation in order to achieve a robust and generic solution for audiobased multimedia indexing and retrieval, specifically: Generic Support for Audio Codecs Generic Support for File Formats Generic Support for Audio Capturing & Encoding Parameters Generic Support for AFeX Framework Parameters The main objective is content-based (speaker, subject, sounds like.. ) retrieval of the audio, which is suitable to human judgment and (aural) perception.

30 Audio Indexing Scheme in MUVIS Audio Stream 1 Classification & Segmentation per granule / frame. Silence Speech Music Fuzzy 2 Audio Framing in Valid Classes Audio Framing & Classification Conversion Uncertain Speech Music Fuzzy AFeX Module(s) 3 AFeX Operation per frame... Speech Music Fuzzy 4 KF Extraction via MST Clustering MST KF Feature Vectors Audio Indexing

31 2. Audio Framing with Classification Conversion Classification per granule/frame M: Music S: Speech X: Silence M M M M M M M M X X X S S S S S S S S S X X Music Uncertain Speech Uncertain Final Classification per audio frame

32 Audio Feature Extraction (AFeX) Framework Independent AFeX module(s) integration capability into MUVIS framework for audiobased indexing and retrieval. DBSEditor AFex_*.DLL AFex_Bind AFex_Init AFex_Extract AFex_Exit AFex_API.h AFex_Bind() AFex_Init() AFex_Extract() AFex_GetDistance() AFex_Exit() AFex_GetDistance MBrowser

33 Key-Framing via MST Clustering S p ee ch L a b 'S' 'ch' 'b' 'ee' 7 p 9 'L' 'a'

34 A Sample AFeX Module Imp.: MFCC MFCC (Mel-Frequency Cepstrum Coefficients) AFeX module provide generic feature vectors independent from the following parameters: Sampling Frequency. Number of audio channels (mono/stereo). Audio Volume level. c i 1/ 2 P π i = (2 / P) = log m cos ( j 0.5) j 1 j N

35 Audio Retrieval in MUVIS Sub-feature Vectors of Query Clip Sub-feature Vectors of a Database Clip FV(0) Matching Class Types FV(i) FV(0) FV(i) For each frame, a search is done to find a matching frame which gives minimum distance. FV(0) FV(i) Feature vectors per class type

36 Audio Retrieval in MUVIS (cont.) In order to accomplish an audio based query within MUVIS, an audio clip is chosen in a multimedia database and queried through the database if the database includes at least one audio feature. Let NoS be the number of feature sets for a database and let NoF(s) is the number of sub-features per feature. Sub-features are obtained by changing the AFeX module parameters or the audio frame size during the audio feature extraction process. D ( s, i f ) = min 0 ( SD( QFV ( s, f ), DFV ( s, f ) ) C C i i j i j C i if if j C j C i i = D( s, f ) = i C i i D i ( s, f ) QD c = NoS NoF ( s) s f W ( s, f ) D( s, f )

37 Experiments in MUVIS Framework: Visual Compression effects on Query Performances

38 Compression effects on Query Performances We used Lossy JPEG image compression method. Why? Widely used, Adjustable quality parameter (compression ratio). We used MPEG-4 and H.263+ video compression methods. Why? Widely used, Allow integrating scene-cut detection algorithms (keyframes for features), Adjustable target bit rate.

39 Features Used in the Experiments We used; HSV and YUV color histograms. Texture features via Gabor Wavelet Transform (GWT) technique. Texture features via Gray Level Co-occurence Matrix (GLCM) technique. Why? Commonly used features in indexing and retrieval area, Practically feasible, reasonable computational complexity, HSV and YUV color spaces are close to human visual system.

40 Experimental Databases 3 base databases created by AVDatabase and Idatabase. Base Image Database: General-purpose database uncompressed color images from several sources. Various sizes (from 100x100 up to 1200x1000 pixels), color bit depths (8-24 bits). Base Rock Texture Image Database: Constraint database uncompressed gray scale texture images of 6 different rock classes. 166x166 pixels size and 8 bits luminance depth. Base Video Database: General-purpose database. 300 video clips from TV channels. CIF-size (352x288 pixels). MPEG-4 compressed with 2 Mbits/sec bit rate. Video clip durations between 40 and 120 seconds.

41 Experimental Databases (cont.) 7 more image databases generated from the base image database. Each one has unique JPEG compression quality (10% to 75%). Together with the base image database, used for HSV and YUV color histogram based experiments. 10 more video databases generated from the base video database. Each one has unique {compression method, bit rate} pair. i.e. {MPEG-4, 1024 Kbits/sec} Used for HSV and YUV color histogram based experiments. 7 more texture databases generated from the base texture image database Each one has unique JPEG compression quality (10% to 75%). Together with the base texture image database, used for GWT and GLCM texture based experiments.

42 Grade Subjective Evaluation of Retrieval Experiments Subjective Meaning 0 No similarity / Not related 1 Slightly related 2 Related 3 Similar 4 Fairly similar 5 Same / Almost identical 12 An evaluation group of 10 people. Performance (P), Query Performance (QP) and Overall Query Performance (OQP) values are calculated. P = G i W where G i [0,5] is the given subjective grade, and W i = i i= 2 (13-i) is the weight for the item in the rank i. NP N Pj QPk where N is the number of queries, and NP (=10) j= k = QP= 1 OQP = 1 NP N is the size of the evaluation group.

A Sample Retrieval Setup Rank Grade 2 5 3 3 4 4 5 0 6 3 7 2 8 1 9 1 10 1 11 1 12 3 P =

43 A Sample Retrieval Setup Rank Grade P = (5*11) + (3*10) + (4*9) + (0*8) + (3*7) + (2*6) + (1*5) + (1*4) + (1*3) + (1*2) + (3*1) = 171

44 Color Based Image Retrieval Performance Results 30 images selected from the base uncompressed database In each database, selected 30 images (compressed with the corresponding compression quality) were queried. Grades were collected to obtain the OQP values. OQP Value Conclusions: The best color based retrieval performance in uncompressed image database Increasing compression ratio gradually decreases the retrieval performance HSV color hist. performs better than YUV color hist. in the uncompressed domain, and vice versa in the compressed domain 100 Base JPEG Compression Quality HSV YUV In general, 30% quality parameter achieves a satisfying compression ratio (94.4%), and does not cause significant loss in the overall retrieval performance.

45 Color Based Video Retrieval Performance Results 15 video clips selected from the base video database In each database, selected 15 video clips (compressed with the corresponding compression bit rate) were queried. Grades were collected to obtain the OQP values. OQP Value Video Compression Bitrate HSV / MPEG-4 YUV / MPEG-4 HSV / H.263+ YUV / H.263+ Conclusions: Color based video retrieval performance is better in MPEG-4 compressed databases. More stable in H.263+ compressed databases at lower bit rates It might be expected that in H.263+ compressed databases, retrieval performance is better at very low bit rates HSV color hist. gives slightly better performance 512 Kbits/sec can be the optimum compression bit rate at high bit rates For low bit rate applications, 128 Kbits/sec for MPEG-4 and 64 Kbits/sec for H.263+ can be preferable

46 Texture Based Image Retrieval Performance Results 15 images selected from the base image database In each database, selected 15 images (compressed with the corresponding compression quality) were queried. Grades were collected to obtain the OQP values. 320 Conclusions: OQP Value Base JPEG Compression Quality Gabor GLCM Texture based image retrieval gives more robust performance results color based retrieval GLCM feature extraction technique gives higher preformance than GWT technique Retrieval performance based on GWT is more stable 25% compression quality achieves the highest compression ratio without loosing significant retrieval performance

47 Conclusion and Remarks MUVIS system is designed to bring a unified and global solution to content-based multimedia indexing and retrieval problem. Furthermore, it supports several user friendly media browsing capabilities. The main objective is to focus on multi-feature support which possibly uses shape, texture and audio along with the color information. MUVIS framework supports integration of the aural and visual feature extraction algorithms explicitly.

48 Conclusion and Remarks (cont.) Following visual features are already implemented and integrated into MUVIS framework: HSV, RGB and YUV Color Histogram BINS, Texture Feature using Gabor Wavelet Transform, Texture Feature using GLCM (Gray Level Cooccurence Matrix). Canny Edge Histogram Feature.

49 Conclusions and Remarks (cont) Audio is important. Sometimes it bears more semantic and content information than visual part. Henceforth the preliminary results shows the effectiveness of the audio-based retrieval compared to visual retrievals (similar or better results). Classification and segmentation algorithm has been recently improved. A new approach based on fuzzyregions and semantic-rule-based classification with intra segment boundary detection has been developed.

50 Conclusions and Remarks (cont.) MFCC prove to be as successful tool for audio based indexing & retrieval as in speech recognition and speaker identification systems. Some problems are still open: Normalization of AFeX feature vectors (for merging). Other AFeX modules should be developed and tested. Studies on merging visual and aural retrievals should be provided to improve overall retrieval performance.

51 Conclusions and Remarks (cont.) MUVIS provides the following advantages via Progressive Query: Query Accessibility Instantaneous Query Retrievals. Query Browsing. Query Stop Capability. The probability to retrieve better (more relevant) results than the final outcome due to the limited sub-set search within the database. Faster retrievals with low memory consumption

DbsEditor v1.8 (Database Editor) QUICK REFERENCE

DbsEditor v1.8 (Database Editor) QUICK REFERENCE By S. KIRANYAZ This application is a dialog-based program, which is designed to perform several off-line tasks of the MUVIS multimedia databases. DbsEditor