SYSTEM PROFILES IN CONTENT-BASED INDEXING AND RETRIEVAL

Transcription

1 1 SYSTEM PROFILES IN CONTENT-BASED INDEXING AND RETRIEVAL Esin Guldogan

2 2 Outline Personal Media Management Text-Based Retrieval Metadata Retrieval Content-Based Retrieval System Profiling User Surveys Analysis of User Surveys CBIR System Profiles and Experimental Cases Experimental Results Conclusions

3 3 Personal Media Recent image technology improvements have led to a huge amount of digital multimedia. Flickr claims to host 3 billion images in November billion video viewed in YouTube on July 2008.

4 4 Personal Media Management Storing, Browsing, Indexing Accessing, Searching, Retrieving Sequential Text-Based Event-Based etc. Content-Based

5 5 Text-Based Search Requires Annotation Indexing Phase Time Consuming Subjective Retrieval Phase Very Fast Supervised Low Accuracy

6 6 Metadata Location, Time, etc. Fast Indexing Fast Retrieval Limited

7 7 Content What is the content of an image?

8 8 Content-Based Indexing and Retrieval Content Based Image Retrieval (CBIR) is a technique of searching through a database of images not based on keywords but image content

9 9 Content-Based Image Retrieval Features CBIR systems analyze image content via features Describe image content using low-level features: color, shape, and texture. High-level features: Red bus, pigeon, rock etc.

10 10 Content-Based Indexing and Retrieval Query Image Feature Extraction Features Display Results Similarity Measurement User ONLINE OFFLINE Images Features Feature Extraction Image Database

11 11 MUVIS Framework SBD Modules SEG Modules FeX Modules AFeX Modules SBD API SEG API FeX & AFeX API Indexing Retrieval Still Images Audio-Video Clips DbsEditor Database Management HCT Indexing MM Insertion Removal MM Conversions FeX - AFeX Management SEG Management SBD Management Image Database Hybrid Database Video Database MBrowser View-Display Query: PQ & NQ HCT Browsing Video Summarization An Image An Audio- Video Clip A Video Frame AVDatabase AV Database Creation Real-time Capturing Encoding Recording

12 12 System Profiling Complex User Interfaces Complex Hardware Dependencies Computational Complexity Efficiency

13 13 System Profiling Tuning and adapting the parameters of the system for improving the performance Increase scalability User Satisfaction

14 14 System Profiling in CBIR CBIR APPLICATION System Profile 1 Indexing Factors / Retrieval Factors / System Profile 2 Parameter 1 Parameter 2 Parameter 1 Parameter 2 Adaptability and Hardware Scalability System Profile 3 Parameter 3 Parameter 3 Parameter 4 Parameter System Profile 4...

15 15 User Surveys in Multimedia Jaimes studied human factors, which influence automatic content-based retrieval systems, such as human memory, context and subjectivity. Eakins, Briggs and Burford used online questionnaire method in order to improve user interface of CBIR applications. Halvey and Keane studied log statistics of YouTube to provide an analysis of user s interaction with video search engines. Frohlich et al. used interview and observation approach in order to understand the strengths and weaknesses of past and present technology of photo sharing. Rodden and Wood used interviews and questionnaires to find out how people organize and browse their digital photo collections. Weiss et. al. studied user-profile based personalization in order to select and recommend content with respect to user s interest for automated online video or TV services.

16 16 User Survey and Participants Identifying real world problems and specifying system requirements and system limitations 122 people contributed to the online survey, 27 females and 95 males participated. students, researchers, engineers and professors from computer science, software systems, electronics, telecommunication, and information technology. Age distribution is as follows: 32% are years old, 61% are years old and 7% of years old.

17 17 Analysis of User Surveys The analysis method of the survey results can be classified into two categories: Direct answers from the question results definitions and specifications of indexing and retrieval parameters Heuristic analysis of the relevant and associated survey questions System profiling a) How would you prefer to organize your multimedia files? By multimedia source 1% Other By location 4% By people 6% 9% By date 34% By events 46% b) What is the reasonable waiting time in your opinion to see the results of an image/video search on the _web_? between 30 sec and 1 min ; 6 % approxim ately 30 seconds; 48 % 1-3 min ; 5 % more than 3 min ; 1 % Instantan eous ; 40 % c) Which of the following do you prefer to see for each multimedia item when browsing? Associate d textual descriptio n (caption, date, file size, Thumbnail ; 64 % Other ; 2 % d) What is the reasonable waiting time in your opinion to see the results of an image/video search on the _homecomputer_? between 30 sec and 1 min ; 7 % approxim ately 30 seconds ; 31 % 1-3 min ; 5 % more than 3 min; 2 % Full-size image ; 30 % Instantan eous ; 56 %

18 18 Survey Results Distinct informative knowledge about the hardware specification of the users and their preferences about digital image management Provide answers to the definition of the system profiles in terms of hardware specifications and technical specifications affecting CBIR parameters and adaptations Helps in the selection of factors, parameters and experimental case setup Example: Answers of the 16th question in the survey reveal that 93% of the participants prefer to use JPEG image compression technique Define the requirements, capacities and conditions of the systems

19 19 System Profiles and CBIR Baseline System Profile General PC and laptop users Powerful System Profile powerful computer systems such as dedicated servers for professional use TV broadcast and mass media companies and, libraries Limited System Profile limited platforms such as mobile phones Distributed System Profile client-server architecture, such as web-based systems Indexing Factors/parameters: Compression parameters Image Downscaling parameters Feature type Retrieval Factors/parameters: Dimension reduction of feature data parameters Feature selection

20 20 Recommended CBIR System Profiles Limited Systems Distributed Systems Baseline Systems Powerful Systems Indexing Factors/ Compression ( JPEG Quality Factor) Image Downscaling Compression quality factor 50% Image Scaling Factor (ISC) = 4 for Color features ISC=2 for texture and shape features Compression quality factor 75-50% Image Scaling Factor = 4 for Color features ISC=2 for texture and shape features Compression quality factor 75% Image Scaling Factor = 2 for Color features none for texture and shape features None or Compression quality factor 90% Image Scaling Factor = 2 or none for Color features none for texture and shape features Feature Factors Use a feature selection method Use a feature selection method Optionally use a feature selection method Optionally use a feature selection method Retrieval Factors/ Dimension Reduction of Feature Data Feature Selection and Combination Scaling factor=4 or 8 Use a feature selection method Scaling factor=4 Use a feature selection method Scaling factor=2 Optionally use a feature selection method None or Scaling factor=2 Optionally use a feature selection method

21 21 Experimental Cases MUVIS Framework Corel Image database 14 types of low-level features YUV, HSV and RGB Color Histograms Dominant Color Feature Gray-level Co-occurrence Matrix Texture Feature Gabor Wavelet Texture Feature Canny Edge Histogram Objective evaluation measurement ANMRR Image Compression JPEG Dimension Reduction of Feature Data Mapping Based Adaptive Threshold (MAT) Image Downscaling DCT-Based

22 22 Experimental Cases System Profiles Limited System Distributed System Baseline System Powerful System Attributes Connection Bandwidth 128/512 Kbit/s 128 Kb/s 1 Mb/s 128 Kb/s 2 Mb/s 1 Mb/s 100 Mb/s Storage Space 1 GB 120 GB client 120 GB 180 GB CPU Power [Information not available] Intel Pentium GHz Intel Pentium GHz 2x2.8 GHz Display Size 320 x x x x 1024 Multimedia Codecs MPEG-4, H.264/AVC, H.263/3GPP, MP3-, AAC-, eaac- and eaac Generally All Generally All Generally All

23 23 Experimental Results Experimental Results of Image Compression Image Compression ANMRR Results Size on Disk Recommended System Profiles Uncompressed GB Powerful System Profile JPEG Compressed with Quality Factor 90% MB Powerful System Profile JPEG Compressed with Quality Factor 75% MB Baseline System Profile Distributed System Profile JPEG Compressed with Quality Factor 50% MB Limited System Profile

24 24 Experimental Results of PSP Compressed Image Database with JPEG Quality Factor 90% Image Downscaling Color-based scaled by 2 & texture and shapebased none Color, texture and shape-based scaled by 2 ANMRR Elapsed Times for Feature Extraction Process on PSP hours hour Compressed Image Database with JPEG Quality Factor 90% AND Images are Downscaled for Feature Extraction Process Dimension Reduction of Feature Data ANMRR Elapsed Times for Retrieval Process on PSP None sec Scaled by sec Scaled by sec Color-based scaled by 4 & texture and shapebased scaled by min Scaled by sec Color, texture and shape-based scaled by min

25 25 Experimental Results of BSP Image Downscaling Color-based scaled by 2 & texture and shape-based none ANMRR Elapsed Times for Feature Extraction Process on BSP hours Compressed Image Database with JPEG Quality Factor 75% Dimension Reduction of Feature Data ANMRR Elapsed Times for Retrieval Process on BSP None sec Compressed Image Database with JPEG Quality Factor 75% Color, texture and shape-based scaled by 2 Color-based scaled by 4 & texture and shape-based scaled by hour hour AND Images are Downscaled for Feature Extraction Process Scaled by sec Scaled by sec Scaled by sec Color, texture and shape-based scaled by min

26 26 Experimental Results of DSP Compressed Image Database with JPEG Quality Factor 75% Image Downscaling Color-based scaled by 2 & texture and shape-based none Color, texture and shape-based scaled by 2 Color-based scaled by 4 & texture and shape-based scaled by 2 ANMRR hours hour hour Elapsed Times for Feature Extraction Process on DSP Compressed Image Database with JPEG Quality Factor 75% AND Images are Downscaled for Feature Extraction Process Dimension Reduction of Feature Data ANMRR Elapsed Times for Retrieval Process on DSP None sec Scaled by sec Scaled by sec Scaled by sec Color, texture and shape-based scaled by min

27 27 Experimental Results of LSP Compressed Image Database with JPEG Quality Factor 50% Image Downscaling Color-based scaled by 2 & texture and shape-based none Color, texture and shape-based scaled by 2 Color-based scaled by 4 & texture and shape-based scaled by 2 ANMRR Elapsed Times for Feature Extraction Process on LSP 0.22 ~65 hours hour hour Compressed Image Database with JPEG Quality Factor 50% AND Images are Downscaled for Feature Extraction Process Dimension Reduction of Feature Data ANMRR Elapsed Times for Retrieval Process on LSP None sec Scaled by sec Scaled by sec Scaled by sec Color, texture and shape-based scaled by hour

28 28 Conclusions and Future Work Novel study for defining CBIR system profiles and determining suitable parameters for each profile substantial savings in time and computational complexities while maintaining semantic retrieval performance Scalable and adaptable CBIR systems Study may be extended and supplemented by additional experiments especially for future CBIR applications and user platforms which are expected to change the proposed profiles and the proposed parameters due to advances in technology. User satisfaction for the proposed system profiles and CBIR parameters using online surveys and further analysis