CERN Open Data and Data Analysis Knowledge Preservation

Size: px

Start display at page:

Download "CERN Open Data and Data Analysis Knowledge Preservation"

Amy Cannon
5 years ago
Views:

1 CERN Open Data and Data Analysis Knowledge Preservation Tibor Šimko Digital Library April / 26

2 2 / 26

3 What is Invenio? digital library and document repository software mature platform: first public release in 2002 rich data: articles, books, notes, photos, videos, software, data some Invenio-based services at CERN: co-developed by an international collaboration participating in EU 3 / 26

4 Data 4 / 26

5 Code you can cite automated GitHub Zenodo bridge push new release to GitHub automatic archival on Zenodo software preserved, minted with a DOI, made 5 / 26

6 Code Data Paper link data (DATAVERSE) to code (ZENODO) to papers (INSPIRE) example: 6 / 26

7 2 7 / 26

8 CERN 8 / 26

9 Large Scale Solutions Primary site: 100k cores (10k nodes), 100k disks (50 PB), 21k NIC Grid: 13 Tier-1 sites, 155 Tier-2 sites, 10 Gbps 9 / 26

10 Preserve 10 / 26

11 Big Data? data scale knowledge raw GB / sec calibration, conditioning reconstructed PB / year filtering, selection reduced TB / analysis user code, physics objects publication GB / analysis correlation, data behind plots... input filtering output... code 11 / 26

12 12 / 26

13 System Architecture TWiki SVN GitHub SharePoint Analysis analysis-preservation.cern.ch file storage abstraction layer CADI CDS INSPIRE... AFS Box Ceph CASTOR Drive 13 / 26

14 Knowledge Representation record format: extended MARC21 technical metadata: beyond bytes e.g. 256 computer file characteristics $a characteristics $e events $t text $b bytes $f files... knowledge metadata: semantics e.g. 505 formatted contents note CSV column information $t title $g miscellaneous internal format: JSON MARC21 JSON schema 14 / 26

15 3 15 / 26

16 Opening Up Data policies: restricted embargo period open [...] Data with high abstraction, such as AOD, will be conditionally made publicly available after an embargo period of 5 years after publication for 10% of the data and 10 years for 100% of the data [...] ALICE Data Policy Challenges: audience: data miners citizen scientists high-school students general public computing: exploring in the browser 16 / 26

17 CERN Open 17 / 26

18 Education 18 / 26

19 Visualise 19 / 26

20 20 / 26

21 Research 21 / 26

22 CMS Primary 22 / 26

23 CernVM 23 / 26

24 Open Data? Who cares? 82,000 distinct users visited the site 21,000 distinct users viewed data records 16,000 distinct users used event display 3,000 distinct users 24 / 26

25 25 / 26

26 CERN (Open) Data Capturing and disseminating knowledge of data, code, platform, processes to enable future data reuse (Open) Data Analysis Preservation Framework CERN IT J. Cowton, P. Fokianos, J. Kunčar, T. Smith, T. Šimko CERN Library S. Dallmeier-Tiessen, P. Herterich, L. Rueda ALICE M. Gheata, C. Grigoras ATLAS K. Cranmer, L. Heinrich, D. Rousseau, F. Socher CMS A. Calderon, A. Huffman, K. Lassila-Perini, T. McCauley, A. Rao, A. Rodriguez Marrero LHCb S. Amerio, B. Couturier, A. Trisovic CERN CernVM J. Blomer CERN EOS L. Mascetti DASPOS M. Hildreth DPHEP 26 / 26

Towards Reproducible Research Data Analyses in LHC Particle Physics

Towards Reproducible Research Data Analyses in LHC Particle Physics Tibor Šimko CERN ILIDE 2017 Jasná, Slovakia 3 5 April 2017 @tiborsimko 1 / 30 Preserving research data @tiborsimko 2 / 30 Why? Scientific