Content-based image retrieval integrated into Fedora

Transcription

1 Content-based image retrieval integrated into Fedora Pierre Yves Burgi IT Division, University of Geneva, Switzerland Patrick Monbaron & Nastaran Fatemi University of Applied Sciences, Yverdon, Switzerland

2 Presentation outline Context MPEG-7 Data migration Indexing Image retrieval Demo Conclusions Perspectives Context MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 2

3 Context of the project Paradigm shift Migration of image collections from Oracle DB to Fedora First step: data synchronization Second step: user interface targeting data retrieval Third step: user interface for ingesting images (through Valet) Why Fedora? Conceptually rich (objects, datastreams, SOA, etc.) Based on open standards (e.g. XML) Convenient for adding datastreams such as MPEG-7 Context MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 3

4 Old (relational) object model Oracle Object Model Objec;fs Context MPEG7 Fedora Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 4

5 New object model Fedora Object Model Fedora s view AUDIT DC MPEG-7 Image_Hi Flux d'audit des modifications de l'objet Flux de métadonnées Dublin Core Flux de métadonnées MPEG-7 importées de la base de données Image haute résolution Image_Low Image basse résolution Thumb Miniature Objec;fs Context MPEG7 Fedora Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 5

6 Why MPEG-7? (and not DC) Makes easier migration from the DB Match from DB s 21 fields to MPEG-7 possible Fits more image description dc:creator versus DS Creator with <Role> & <Agent> <VisualDescriptor>, <MediaFormat>, etc. Exif metadata Etc. Context MPEG 7 MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 6

7 What is MPEG-7? Context MPEG 7 MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 7

8 MPEG-7: feature extraction Edge histogram descriptor Scalable color descriptor Color layout descriptor Caliph and Emir Context MPEG 7 MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 8

9 MPEG-7: encoding Context MPEG 7 MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 9

10 MPEG-7: retrieving Match is expressed as a number Context MPEG 7 MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 10

11 Data migration Oracle DB XSLT Record reading XML MPEG7 + FOXML Data migration FOXML Image Applica;on server Fedora API-M Messagin g FOXML gsearch XSLT indexing File system Lucene Context MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 11

12 Data synchronisation 3 Phases Delete obsolete objects «Delete» in Fedora those recordings which do not exist anymore in Oracle DB Update objects Update in Fedora objects corresponding to recordings modified in Oracle DB Create new objects Create new objects in Fedora corresponding to the new recordings present in Oracle DB Algorithm based on the date of the last batch Date saved within a configuration file Context MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 12

13 Migration: delete objects Reading of the id of all Oracle recordings Reading of the id of all Fedora objects. Compare both lists and «delete» from Fedora missing elements in Oracle Context MPEG7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 13

14 Migration: create new objects Read date of last object creation Search id of those elements created since last update For new objects: Reading of metadata and image analysis Data migration into XML format and into MPEG-7 and FOXML through XSLT Ingest FOXML and images into Fedora Context Fedora Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 14

15 Migration: update objects Read date of last object creation Search id of those elements updated since last update Reading of metadata and image analysis Data migration into XML format and into MPEG-7 Ingest MPEG-7 into Fedora Context Fedora Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 15

16 Indexing Indexing of DC Textual MPEG-7 metadata Not indexed but saved in the index Technical MPEG-7 metadata Image attributes Context Fedora Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 16

17 Indexing Indexing process within Lucene Oracle DB XSLT Record reading XML FOXML Application server Fedora MPEG7 + FOXML Data migra;o n Image API M Messagin g FOXML gsearch XSLT Indexing File system Lucene Context Fedora Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 17

18 Image retrieval Application server Fedora API A gsearch XSLT User interface Request parsing HTML page genera;on XSLT Resolver Indexing gsearch connector Search by image matching File system Lucene Objec;fs MPEG 7 Migra;on Indexing Retrieval Demo Conclusions Perspec;ves 18

19 Retrieval by image matching Image reference id Retrieval of the attributes of this image from Lucene Match of each image attribute with that reference and computation of a matching score Selection (10 first) Reading of all data from Lucene and conversion into XML XML output for display Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 19

20 Demo: Photothèque Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 20

21 Conclusions Applications of the MPEG-7 standard to image matching retrieval give satisfactory results but this is not terribly semantic! For large image data bank indexation method might not be optimal (now about images) Parameter tuning of image attributes not easy Image corpus too small to establish benchmarks Migration procedure from a DB to Fedora tested Data synchronization remains a difficulty Integration within Fedora otherwise is a success Culture change (paradigm shift) in progress Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 21

22 Perspectives Exploiting more Fedora s features Using disseminator for watermarking images on the fly Applying XACML policies Improving user interface Play with search parameters (shape versus color) Applying the methodology to other image data banks Medicine, architecture, etc. Other media (audio, video) Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 22

23 Questions Complete report available in French at: Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 23

24 «building» (key word) Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 24

25 Retrieval results Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 25

26 Display of chosen image Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 26

27 Retrieval by image matching Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 27

28 Retrieval results (by contour matching) Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 28

29 Display of an other chosen image Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 29

30 Retrieval results (by color matching) Context Fedora Migra;on Indexa;on Retrieval Demo Conclusions Perspec;ves 30