AGGREGATIVE DATA INFRASTRUCTURES FOR THE CULTURAL HERITAGE

Transcription

1 AGGREGATIVE DATA INFRASTRUCTURES FOR THE CULTURAL HERITAGE Max Planck Institute, 13 th of March, 2013 Paolo Manghi Istituto di Scienza e Tecnologie dell Informazione, Consiglio Nazionale delle Ricerche, Area di Pisa

2 Istituto di Scienza e Tecnologie dell Informazione (ISTI) Consiglio Nazionale delle Ricerche

3 InfraScience Group Head of research: Donatella Castelli From: Digital Libraries: data models and services To: Data Infrastructures for Science and Scholarly Communication

4 Aggregative Data Infrastructures (ADIs) Information systems where communities and their organizations (e.g. research centers, universities, industries) find tools for Aggregate heterogeneous data sources to form uniform (common data model) and (possibly) richer information spaces Mainly act at the level of metadata descriptions, to enable cross-data source, cross-discipline, crosscommunity access services and leverage the visibility of participating data sources

5 ADIs and Cultural Heritage (CH) The Cultural Heritage domain most active in the realization of ADIs Increased and continuous availability of data sources with CH digital content calls for immediate and integrated access to such content CH communities set up ADIs Europeana CLARIN Linked Heritage ATHENA EFG/EFG1914 EAGLE HOPE

6 The Heritage of People s Europe Satellite project of Europeana, with focus on European social and labor history from 18 th to 21 st century Access services for cross-country, cross-discipline, cross-institution opportunities HOPE ADI Data Sources from CH institutions Sub-communities (multidisciplinary) Europeana Data Model Information Space HOPE Uniform Information Space (HOPE data model) Library Archive Visual Audio/Video

7 Data sources HOPE ADI requirements: data collection and conversion CH institutions archive library a/v library visual Export Models LIDO DC EAD Propr. EN Access Protocols ODBC OAI FTP OAI Mapping Data source profiles: library, archive, visual, audio/video HOPE Common Data Model a/v visual DC OAI MPEG21 SRW

8 HOPE Data Model

9 HOPE ADI requirements HOPE Information Space Synchronized with data source metadata Provision: Web portal: search individual entities, navigate relationships between entities Standard access protocols for third-party services Push files towards social network (e.g. YouTube, Flickr) Enhancement: Manual bulk-tagging of objects (vocabulary of historical themes) Manual editing of aggregated records EDM Information Space Synchronized mirror of the HOPE information space Provision: Available via OAI-PMH

10 HOPE numbers HOPE ADI information space: Today: metadata records (one per file) from 12 content providers, available via Europeana Final threshold: about metadata records, describing around files, available from the IHALI portal and Europeana

11 Outline What we mean by data in ADIs Data modeling and data representations ADI Information Spaces Definition Mirroring and staging Realizing ADIs Data conversion Data enhancement ADI high-level functional architecture Technological and sustainability issues D-NET Software Toolkit D-NET Software Toolkit Real-case: HOPE project

12 What we mean by data in ADIs Content type: Files (corpora): payloads, reachable via the Internet and interpretable by applications Metadata: descriptions of one or more files, for the purpose of discovery and/or reuse by humans and/or machines Metadata: data modeling and representations Conceptual model, e.g. MARC, DC, EAD, LIDO, EDM, proprietary Physical representation (encoding conceptual model), e.g. relational database, full-text index, native XML DB, file system with XML files, Excel files, triple stores Export models, e.g. MARC, DC, EAD, LIDO, EDM Export formats (encoding export model) and protocols, e.g. OAI-PMH/XML, OAI-ORE/XML-RDF, LinkedData/SparQL-RDF

13 Operating data sources: conceptual and physical data models Conceptual model and objects E-R publication Title Date DOI P 1:N 1:N Name author file URL License A1 A2 F1 F2 Mapping from conceptual to physical data models Relational Model Physical representation Table publication Table file Table author Title Date DOI: PK URL: PK License Publication: FK Name PK Publication: FK P: The Divine Comedy of Errors, 2013, doi:12345 A1: W. Alighieri, wa@florence.it, ref-to P A2: D. Shakespeare, ds@london.it, re-to P F1: http: a, OpenAccess, refto P F2: http: b, Copyright, refto P

14 Data Sources: export data models and formats Name author Conceptual Model 1:N publication 1:N Relational Database Title Date DOI URL License file Export model Dublin Core (flat) data model publication dc:title dc:date dc:creator Export format and protocol OAI-PMH/ XML LinkedData/ RDF JSON, AVRO Logical mapping between conceptual data models Mapping from physical representation to export formats Implementation of access protocol APIs

15 ADI Information Spaces Information Space Container of data conforming to a conceptual model and encoded according to a physical representation It exposes its content according to one or more export models and relative access protocols Input requirements: enhancement/provision functionalities Stores its data according to one main physical representation Information Space can have mirrors Clones (replicas): copies of the data stored according to the same physical representation Twins: copies (or subsets) of the data stored according to different physical representations Information Spaces can be staged into other (different) Information Spaces Caching Information Space (optional) Model-less information space, used to cache all data collected from data sources according to their export format

16 Operating ADIs: an example Data collection and Information Space mirroring Data Sources EM EFM:XML EM EFM: XML Logical mappings A P I A P I Collection Synchronization Information Space Conceptual model ADI PM: XML files ADI PR: full-text index ADI PR: triple store ADI A P I Conversion Synchronization EM: export model PR: physical data model Transformation Cleaning Scalable fulltext Search SparQL access services

17 Operating ADIs: an example Information Space Staging Logical mappings CM: conceptual model PM: physical representation Information Space CM A Information Space CM B ADI PR: XML files A P I Conversion Synchronization ADI PM: relational database A P I ADI PR: full-text index Conversion Synchronization ADI PR:. Conversion Synchronization ADI PR: triple store ADI PR:.

18 Realizing ADIs: Information Space Conversion Managing data mappings Structure: transformation Semantics: normalization/cleaning Managing synchronization in mirroring and staging Handling specific mirroring and staging dependencies Managing identifiers across Information Spaces Generating identifiers De-referencing identifiers Reification of identifiers from metadata information

19 Realizing ADIs: Information Space Conversion Mappings map entities and relationships from export models (i.e. data sources, information spaces) onto entities and relationships of Information Space data models Entity property-values may transform into an entity with a relationship to a new entity type, which contains a property with the same value Origin CM mapping Information Space Conceptual model publication dc:title dc:date dc:creator E-R 1:N publication 1:N Title Date DOI dc:creator -> Name author file URL License The implementation of the mappings depends on the export format and on the physical representation of the Information Space data model

20 Realizing ADIs: Information Space Conversion - examples Export formats: XML Information Space physical representation: XML Simple case (1:1 mapping): one XML embedding all information Complex case (1:N mapping): each entity in the target model corresponds to a different XML schema, entities may contain relationships to other entities Information Space physical representation: relational databases Normalized relational database: scripts to un-package XML records and create the associated records and relationships Export formats: LinkedData Information Space physical representation: XML (different granularity) or relational databases Require data collection/caching first, then packaging scripts to create XML records or relational db records

21 Realizing ADIs: Information Space Enhancement - example Information Space Conceptual model Enhancement actions ADI PM: XML files A P I Metadata Editor ADI PR: triple store Annotation Inference by mining ADI PR: full-text index Scalable fulltext Search

22 ADIs: abstract functional architecture Data Enhancement Area Editing, enrichment, validation, deduplication, annotation, inference by mining, quality, etc. Portals, Standard access APIs, pushing files, etc Data Provision Area Data Storage and Indexing Area Physical Data Model(s) Physical Data Model(s) Information Space(s) Collected Files Collected Metadata Transformation, Cleaning, identifier assignment, feature extraction, etc. Data Conversion Area Data Media)on Area Data Collection Data Source Management External Data Sources Metadata Repositories File Repositories

23 Realizing ADIs: technological issues Data mediation Handling a dynamic set of data sources Handling interoperability with different access protocols, e.g. OAI- PMH, FTP, ODBC/JDBC, SRW, HTTP Handling history of data collections Data storage and indexing area Handling management and integration of several technologies, e.g. relational DBs, triple stores, column stores (NOSQL), native XML, file systems, full-text indices

24 Realizing ADIs: technological issues Data conversion Handling mapping management for many data sources and coordinating data collection with data transformation, e.g. catalogue of XSLTs, scripts in some programming language Handling quality measurement and validation controls of original data and output data, e.g. metadata quality measures and tools Data enhancement Developing tools for manual or automatic data enhancement Handling storage/preservation and history of data enhancements, e.g. merging similar records, editing records, deleting records, interlinking records Data Provision Handling custom search, browse, and export of metadata with respect to the different Information Spaces and possible physical representations

25 Realizing ADIs: technological issues Gluing functionalities together Building data processing workflows (e.g. collection, mirroring, staging, application of enhancements) by making different technologies interoperate Keeping everything under control Tools for building, executing and monitoring workflows Addressing and monitoring Quality of Storage scalability Processing power Availability of Robustness (resilience to failure, e.g. replicas, clone services) Real-time requirements (e.g. end-user edits) Data quality

26 Realizing ADIs: sustainability issues Data sources lack of funds and are reticent to join yet another ADI (Why should I do it? What s the guarantee of a benefit?) Communities do often lack of funds to operate, maintain ADIs and let them evolve according to the dynamic functional requirements of the community

27 Realizing ADIs: sustainability issues Trade-off cost vs. community satisfaction Designers, developers, administrators, data curators Initial cost: realizing information systems addressing ADI functional and QoS requirements Refinement cost: addressing new requirements, i.e. evolution of science and technology Designers, developers: new access protocols, new storage systems, new functionalities Administrators: variations in QoS Data curators: new data sources, new data Choosing the proper software technology is crucial for ADI sustainability

28 D-NET SOFTWARE TOOLKIT SUSTAINABLE AGGREGATIVE DATA INFRASTRUCTURES

29 D-NET Software Toolkit Designed to facilitate designers and developers in the realization and maintenance of ADIs Open source (web)service-oriented framework

30 D-NET Software Toolkit D-NET service properties Customizability: can be configured to treat data of given data models Modularity: conceived to be part of data processing workflows Data exchange APIs (i.e. ResultSet protocol) Subscription & notification APIs Blackboard protocol APIs (ESP-like asynchronous messaging) Extensibility: further services can be added to introduce new pipeline-able functionalities D-NET ADI properties: Autonomicity, distribution, sharing, to enable the operation of autonomic, scalable and robust ADIs.

31 Data Enhancement Area Hadoop De- duplica)on Metadata Editor Record Tagging TextMining Inference Hadoop OAI Publisher SRW/CQL Publisher Generic portal service Social Network Publishing Data Provision Area Data Storage and Indexing Area MongoDB (GridFS) Postgres Object Store Database Full- Text Index Metadata Store Solr (sharded) MongoDB Column Store HBASE Feature Extrac)on Metadata Cleaner Metadata (un)packaging Metadata Transforma)on XSLT Groovy Data Conversion Area Data Media)on Area FTP Import OAI- PMH Harvester Data Source Man Data Source Validator External Data Sources Metadata Repositories File Repositories

32 Enabling services ExistDB Informa)on Manager ChronJob Sarasvati ResultSet PID AAI Information : resource (de)registration, discovery subscription&notification Resources are: services configurations, workflows, mappings, factorypattern data structures, etc. Manager : workflow engine ResultSet : file exchange protocol Chron Job : scheduler PID : PID generation and de-reference

33 The HOPE ADI Access services for cross-country, cross-discipline, cross-institution opportunities HOPE ADI Data Sources from CH institutions Sub-communities (multidisciplinary) Europeana Data Model Information Space HOPE Uniform Information Space (HOPE data model) Library Archive Visual Audio/Video

34 HOPE: D-NET Configuration Seven Information Spaces: XML physical representations Caching Information Space Inter-profile Information Spaces: MARCXML conceptual model for libraries, EAD conceptual model for archives, EN conceptual model for audio video LIDO conceptual model for visual HOPE Information Space EDM Information Space

35 HOPE: D-NET workflows LMF (Local Metadata Format): data source metadata format PMF (Profile Metadata Format): metadata standard chosen for the profile HMF: HOPE Metadata Format EDM: Europeana Metadata Format Intra-profile workflow LMF LMF PMF Data OAI- PMH Metadata Metadata Metadata Store Metadata Store Source Harvester (un)packaging Transforma)on Cross-profile workflow Full- Text Index HMF Metadata Store Metadata Cleaner HMF-structure Metadata Store Metadata Transforma)on Metadata (un)packaging Social Network Publishing Tagging Tool ContentChecker Metadata Transforma)on Metadata Store EDM Europeana workflow

36 Workflows autonomic management Add new data source to the ADI Register the data source specifying one of the four workflow profiles Provide the mapping of the format onto the standard profile Configure chron job for data collection (refresh or incremental) Data synchronization Event-based workflows: metadata harvesting fires sequence of conversions Data quality (manual) Mapping refinement: check quality of results before committing to the index (manually firing the workflows until done)

37 Acknowledgments (start in April 2013)