Heiðrun. Building DPLA s New Metadata Ingestion System. Mark A. Matienzo Digital Public Library of America

Size: px

Start display at page:

Download "Heiðrun. Building DPLA s New Metadata Ingestion System. Mark A. Matienzo Digital Public Library of America"

Shana Gaines
5 years ago
Views:

1 Heiðrun Building DPLA s New Metadata Ingestion System Mark A. Matienzo <mark@dp.la> Digital Public Library of America Metropolitan New York Library Council Annual Conference January 15, 2015

2 Outline 1. Original DPLA Infrastructure 2. The DPLA ingestion process 3. Challenges with ingestion 4. Feedback from DPLA Hubs 5. Planning for needed improvements 6. Building Heiðrun

3 Original Infrastructure Frontend (Ruby on Rails) API (Ruby on Rails) PostgreSQL Ingestion system (Python) CouchDB River Elasticsearch

4 The DPLA Ingestion Process

5 Metadata Application Profile edm:webresource edm:hasview ore:aggregation dcmitype:collection edm:aggregatedcho dcterms:ispartof dpla:sourceresource dcterms:spatial dcterms:temporal dpla:place edm:timespan

6 Ingestion workflow Harvest hub metadata Transform to DPLA MAP Enrich metadata to clean/add value Save/index enriched metadata Remove deleted records Perform QA if needed

7 Transformation & enrichment Sample pipeline for Portal to Texas History

8 Challenges with ingestion Ingestion process very hands-on; requires significant staff time despite use of common standards Ingestion process not modular and flexible enough to support partial reharvesting or enrichment System has lack of awareness of MAP data as RDF Some enrichment processes (e.g. geocoding) introduce and expose metadata inconsistencies Unqualified Dublin Core requires the most work in terms of mapping and transformation

9 Feedback from DPLA Hubs Greater control over and feedback during the ingestion process Access to data quality reports Provide mechanism to receive enrichments applied by DPLA ingestion process Collaborate on further tool and infrastructure development

10 Planning for improvements Improvement of documentation for metadata model and ingestion process Revision of the DPLA Metadata Application Profile Reassessment of data quality and validation in the context of DPLA Encouraging Hubs to undertake metadata transformation and enrichment locally and to develop appropriate tools Replacement of the DPLA ingestion system

11 Building Heiðrun DPLA started development on new ingestion system and metadata repository in October 2014 Collaborative project across both DPLA Content and Technology teams

12 Development goals Make it easier to harvest and map metadata from various sources/schemas into DPLA MAP Improve enrichment using external sources Actively involve partners in ingestion process through better tools Native support for DPLA MAP as RDF data model

13 Current features Improved harvesting, including support for partial harvests Domain-specific language for metadata mapping Improved scoping of enrichments as field- or recordbased Basic QA environment

14 Future plans Ingest dashboard for DPLA and hub staff Improved QA tools and reports Browser-based GUI metadata mapping tool Building an aggregation system in a box for use by DPLA hubs and others More control for both DPLA Content Team and Hubs staff

15 Thank You! Mark A. Matienzo Digital Public Library of America This work is licensed under a Creative Commons Attribution 4.0 International License.

16 Heiðrun Architecture QA and management interface Generic (shared) enrichments Hub-specific enrichments and enrichment profiles Generic (shared) metadata mappings Hub-specific metadata mappings Krikri Harvesters Harvester settings for specific hubs Heiðrun DPLA MAP models ActiveTriples RDF.rb Apache Marmotta RDBMS (PostgreSQL or MySQL) Solr/Elasticsearch

17 Resources DPLA ingestion system ( legacy system). DPLA new ingestion system code bases. Matienzo, Mark A. and Rudersdorf, Amy. The Digital Public Library of America Ingestion Ecosystem: Lessons Learned After One Year of Large-Scale Collaborative Metadata Aggregation. Proc. Int. Conf. on Dublin Core and Metadata Applications, view/3700.

What s New in DPLA Technology? Mark A. Matienzo Director of Technology, DPLA DPLAFest Indianapolis, IN April 17, 2015

What s New in DPLA Technology? Mark A. Matienzo Director of Technology, DPLA DPLAFest Indianapolis, IN April 17, 2015 Ingestion Workflow Overview Heiðrún: DPLA s new ingestion system Motivation for development/overall