BIDS 2016 Santa Cruz de Tenerife

Size: px

Start display at page:

Download "BIDS 2016 Santa Cruz de Tenerife"

Pierce Conley
5 years ago
Views:

1 BIDS 2016 Santa Cruz de Tenerife 1

2 EUCLID: Orchestrating the software development and the scientific data production in a map reduce paradigm Christophe Dabin (CNES) M. Poncet, K. Noddle, M. Holliman, M. Melchior, O.R.Williams, A. Belikov, J. Hoar on behalf of Euclid Science Ground Segment System Team BIDS 2016 Santa Cruz de Tenerife 2

The Euclid Mission M2 mission in the framework of the ESA Cosmic Vision Programme Euclid mission objective is to map the geometry and understand the nature of the dark Universe (dark energy and dark

3 The Euclid Mission M2 mission in the framework of the ESA Cosmic Vision Programme Euclid mission objective is to map the geometry and understand the nature of the dark Universe (dark energy and dark matter) Actors in the mission: ESA and the Euclid Consortium (institutes from 14 European countries and USA, funded by their own national Space Agencies) Euclid Consortium: 15 countries 100+ labs members Biggest collaboration! For more information see : BIDS 2016 Santa Cruz de Tenerife 3

4 Glossary EAS : Euclid Archive System EAS-DPS : Data Processing System EAS-SAS : Science Archive System EAS-DSS : Data Storage System IAL : Infrastructure Abstraction Layer COORS : Data Processing and storage orchestration M&C : Infrastructure Monitoring and Control CODEEN : Collaborative Development Environment EDEN : Euclid development Environment (common libs) LODEEN : Local Development Environment (VM) SDC : Science Data Center OU : Organisational Unit (Algorithm ad validation) SOC : Science Operation Center MOC : Mission Operation Center PLM : Payload Module SGS : Science Ground Segment SWG : Science Working Group (Science requirements) SVM : Satellite and Service Module VIS: Euclid Instrument Visible NISP : Near Infrared Spectro Image BIDS 2016 Santa Cruz de Tenerife 4

5 Euclid Mission at a Glance BIDS 2016 Santa Cruz de Tenerife 5

6 Euclid Ground Segment BIDS 2016 Santa Cruz de Tenerife 6

Euclid Processing Overview VIS, NIR, EXT: production of fully calibrated photometric exposures from Euclid and groundbased Surveys SIR: production of fully calibrated 1D spectra extracted from the

7 Euclid Processing Overview VIS, NIR, EXT: production of fully calibrated photometric exposures from Euclid and groundbased Surveys SIR: production of fully calibrated 1D spectra extracted from the NISP spectroscopic exposures. MER: production of a source catalog containing consistent photometric and spectroscopic measurements. PHZ: production of the photometric redshift for all catalogued sources. SPE: production of spectroscopic redshifts for all sources with spectra. SHE: measurements of galaxy shapes. LE3: production of all high-level science products. SIM: production of all the simulated data necessary to validate the data processing stages, and to calibrate observational or method biases. BIDS 2016 Santa Cruz de Tenerife 7

8 Architecture key concepts No Dedicated Processing SDC: Any pipeline should run on any SDC (with some exceptions, e.g. Level 1, EXT ingestion, LE3?) Distributed Data and Processing Each SDC is both a processing and a storage «node» Move the code, not the data Run the pipeline where the input data (sky allocation) is stored Separation of metadata (inventory) from data (storage) Map/Reduce analogy Lower level of processing LE2 on single Observation (or set of Observations with overlap) constituting catalogs of sources Higher level of processing LE3 based on global catalogs of sources properties and massive use of simulation (Monte Carlo) for 2 point correlation function BIDS 2016 Santa Cruz de Tenerife 8

9 Architecture : move the code not the data soc SIM VIS SIR NIR MER EXT SPE PHZ SHE LE3 EAS Pipeline in a Box SDCs dedicated to some Processing stages Pipeline in a box on any SDC Unequal load between SDCs How to deal with a new SDC? How to deal with the loss of an SDC? Each SDC = SPOF How to set up and fund redundancy? Data Centric rather than a Processing Centric approach is more relevant: Allocate the data and not the processing to the SDCs Run AMAP the whole pipeline (up to LE2) on any SDC on the smallest meaningful processable bundle of data (QoD: Quantum of Data) Data Volumes over WAN (PBs)! BIDS 2016 Santa Cruz de Tenerife 9

10 Logical Architecture A set of Services which allows a low coupling between SGS components : e.g. metadata query and access, data localization and transfer, data processing, M&C, IAL (science agnostic) A Euclid Archive System (EAS) A central Metadata Repository (DPS) which inventories, indexes and localizes the huge amount of distributed data, A Distributed Storage System (DSS) of the data over the SDCs (ensuring the best compromise between data availability and data transfers), with redundancy M&C and Orchestration (COORS) layers responsible for distributing data and processing among the SDCs, according to a distribution policy An Infrastructure Abstraction Layer (IAL) allowing the data processing software to run on any SDC independently of the underlying IT infrastructure, and simplifying the development of the processing software itself (e.g. takes care of I/O and I/F) BIDS 2016 Santa Cruz de Tenerife 10

Data Distribution : Map on Observations Observations with

to SDC s 600 GB/day Daily reprocessing 10 TB / day Number of

resources LE2 products Catalog : photometry astrometry spectro

11 Data Distribution : Map on Observations Observations with overlaps are colocated on a given SDC Daily distribution SOC to SDC s 600 GB/day Daily reprocessing 10 TB / day Number of Observations per SDC function of SDCs computing/storage resources LE2 products Catalog : photometry astrometry spectro redshift photo redshift shear morphology.. BIDS 2016 Santa Cruz de Tenerife 11

Reduce on global catalogs LE3 processing stages (monthly cadence TBC) Data Flow between SDCs : 20 GB/Day (TBC) Processing needs 200 000 core x hour (TBC) Science ready products (more than 45!

12 Reduce on global catalogs LE3 processing stages (monthly cadence TBC) Data Flow between SDCs : 20 GB/Day (TBC) Processing needs core x hour (TBC) Science ready products (more than 45!) : 2-point statistics for galaxy clustering and weak lensing Mass mapping, 3-point statistics, and galaxy clusters detection Milky Way and Nearby Galaxies Transient phenomena BIDS 2016 Santa Cruz de Tenerife 12

with Elastic Search and Kibana IAL : Pydron CODEEN : Other technologies to

13 Technologies of the components EAS DPS : Object Oriented DB talks to come (RuG / ESAC) M&C : Network monitoring SDC resources monitoring Logs storage with Elastic Search and Kibana IAL : Pydron CODEEN : Other technologies to be explored especially for catalog storage BIDS 2016 Santa Cruz de Tenerife 13

14 EuclidVM principles Any Euclid Processing Function (PF) should run on any SDC, but the SDCs are not homogeneous: Hosting O/S, compilers, libs and versions, Kind of infra: cluster, cloud, shared storage or not, Thus concept of EuclidVM A Processing node VM appliance for Euclid Relies on virtualization at any SDC: independence from hosting O/S Allows to deploys the same guest processing VM everywhere Develop, test, integrate, validate once on a reference platform EuclidVM: Lightweight VM with core O/S (CentOS 7) and most stable core S/W Dynamic Deployment of libs and PF S/W in push or pull mode Candidate technos CernVM ecosystem (µcernvm, CernVM-FS, ) Docker BIDS 2016 Santa Cruz de Tenerife 14

15 Continuous Deployment Need to symplify chain from development to deployment (DEVOPS) allow early tests & improvements on target SDCs Candidate Solution: CernVM-FS Principles: A central repository of Software (Stratum 0) => unique reference A set of distributed replicas (Stratum 1) => scalability and availability SDC Local Squid proxies => performances CernVM-FS client installed at each Processing Nodes Optimized HTTP protocol Local cache Files are downloaded and cached only on access BIDS 2016 Santa Cruz de Tenerife 15

16 Continuous Deployment : cernvm SW files accessed in Run Time, then kept in cache, and released when full BIDS 2016 Santa Cruz de Tenerife 16

17 Development Environment DEVOPS principles: continuous integration & deployment CODEEN (Collaborative Development Environment) eplatform based on Jenkins engine allowing continuous integration and deployment approach Steps: Build, Doc., Tests, Quality Check, Packaging, Deployment LODEEN (Local Development Environment) Ready to use VM dedicated to S/W development on local machine Standards (EDEN) O/S : SL6, => CentOS 7 Languages: C++ / Python Restricted set of supported libs Coding standards BIDS 2016 Santa Cruz de Tenerife 17

18 Development Environment - EDEN BIDS 2016 Santa Cruz de Tenerife 18

19 Conclusions Big challenges to address Validation of the Architecture model Assess the robustness of the key components : EAS / IAL Assess the scalability of the overall architecture and key components Prototyping activities are on going and allow some intensive tests campaigns Loosely coupling components should allow to switch to new technologies Validation of the Pipeline Model and the main figures : data IO and computing resources with scientists Computing resources Architecture should be flexible and scalable to host new Computing Centers if needed Cloud services (processing on demand) could be an opportunity that could be explored Explore optimization / parallelization of key algorithms with scientists BIDS 2016 Santa Cruz de Tenerife 19

the Euclid SGS developement inside ESA and EC, too

20 Thank you for your attention Acknowledgments: authors are indebted to all the indididuals participating in the Euclid SGS developement inside ESA and EC, too many to be listed here BIDS 2016 Santa Cruz de Tenerife 20

Using CernVM-FS to deploy Euclid processing S/W on Science Data Centres

Using CernVM-FS to deploy Euclid processing S/W on Science Data Centres M. Poncet (CNES) Q. Le Boulc h (IN2P3) M. Holliman (ROE) On behalf of Euclid EC SGS System Team ADASS 2016 1 Outline Euclid Project