Preparing Digital Collections for Big Data Analysis. Sven Schlarb, Austrian Institute of Technology e-archiving, Cordoba, Spain 05 th October 2018

Transcription

1 Preparing Digital Collections for Big Data Analysis Sven Schlarb, Austrian Institute of Technology e-archiving, Cordoba, Spain 05 th October 2018

2 Digital Transformation Copyright Doc Searls,

3 Digital Transformation Copyright (network diagram) CC BY-SA 4.0

4 4 Archiving at internet scale

5 5 Is big data still a hype? 2014 BIG DATA Jeremykemp at English Wikipedia [GFDL ( or CC BY- SA 3.0 ( from Wikimedia Commons 05/10/2018

6 6 Is big data still a hype? 2015 Jeremykemp at English Wikipedia [GFDL ( or CC BY- SA 3.0 ( from Wikimedia Commons

7 7 Is big data still a hype? 2018 BIG DATA Jeremykemp at English Wikipedia [GFDL ( or CC BY-SA 3.0 ( from Wikimedia Commons

8 8 To SQL or to NoSQL? Relational databases NoSQL databases

9 Different Nosql database types K1 AAA,BBB,CCC K2 K3 K4 K5 AAA,BBB AAA,DDD AAA,2,01/01/2018 3,ZZZ,5623 Key-Value Wide Column Key Participant Conference ID Name City Name Address City 1 John London PVC2018 Townroad 2 Manchester 2 Linda Palme TFC2018 Market 2 Berlin NoSQL Databases { "name": "Sven Schlarb", " ": "sven.schlarbait.ac.at", "events": [ Event Graph Document { "name": "Kulturhackathon openglam.at", "date": " T00:00:00.000Z" Person }, { "name": "e-archving Cordoba", Person "date": " T00:00:00.000Z" } ] }

10 E-ARK Experimental Cluster Task Trackers Job Tracker Name Node CPU: 2 x 2.40GHz Quadcore CPU (16 HyperThreading cores) RAM: 24GB DISK: 3 x 1TB DISKs configured as RAID5 (Redundanz) 2 TB effective Data Nodes CPU: 1 x 2.53GHz Quadcore CPU (8 HyperThreading) RAM: 16GB DISK: 2 x 1TB DISKs configured as RAID0 (Performance) 2 TB effective Of 16 HT cores: 5 for Map; 2 for Reduce; 1 for OS. 25 processing cores for Map tasks 10 processing cores for Reduce tasks

11 Package transformation and Ingest Reference Implementation Full-text indexing & search Modular package transformation workflows & metadata creation Parallelize full-text indexing Access Fast random access to individual files Faceted Search & Data Mining Aggregating data using facet queries Data mining (Classification, NER)

12 E-ARK Information Package (simplified) SIP representations Metadata edits Migrations Add emulation info DIP metadata Structural metadata Provenance metadata Descriptive metadata Technical metadata SIP DIP [schemas/documentation]

13 earkweb earkweb is based on Phython and the Celery task execution system. Create archival workflows from predefined tasks which can be executed in parallel on a computer cluster. Examples are data validation, format migration, content extraction, database transformation, packaging, interfacing with storage systems. earkweb provides a graphical interface and can be used interactively as well as in batch mode.

14 Cluster Deployment Stack Information package status Task results <<search and retrieval>> decoupled <<notification>> Worker Worker Worker Worker Staging/Storage Area NAS <<package transfer>> 6/30/16

15 Standalone Deployment Stack Information package status Task results <<search and retrieval>> Worker Worker Worker Worker Staging/Storage Area NAS <<indexing>> 6/30/16

16 Data Mining/NLP Purpose: Analyse digital resources of collections Selected use cases: Location names occurring in texts. Named entity recognition and incorporation of geoinformation Text classification

17 Location names occurring in texts StanfordNER for NER nominatim (database behind openstreetmap.org) for georeferencing peripleo for visualization

18 Location names occurring in texts Peripleo - PELAGIOS Project

19 Geographical/timeline search Provided: GML data and TIFF images of maps with metadata (coordinate system, time, etc.) Convert GML data to Peripleo RDF Translate coordinate system if necessary Use peripleo to search for and visualize regions and filter by time Peripleo - PELAGIOS Project

20 Geographical/timeline search Peripleo - PELAGIOS Project

21 Text classification using scikit-learn Prepare data to train SVM classifier Dump full-texts of the repository into reusable packages Apply text classification and update SolR records accordingly

22 Database archiving, rebuilding and analysis e.g. Postgres SIARD e.g. Oracle RDBMS data (up to 80TB) Submit... Archive... Reconstruct... Analyse. source: wikipedia

23 Muchas Gracias por su atención! Hay preguntas?