Euclid Consortium The Euclid Challenges. Pierre Dubath SDC-CH Department of Astronomy University of Geneva. IAU Symposium 325.

Size: px

Start display at page:

Download "Euclid Consortium The Euclid Challenges. Pierre Dubath SDC-CH Department of Astronomy University of Geneva. IAU Symposium 325."

Sybil Greer
5 years ago
Views:

1 The Challenges Pierre Dubath SDC-CH Department of Astronomy University of Geneva IAU Symposium 325 Astroinformatics Sorrento (Italy), October 20-24, 2016 data processing challenges 1

2 List of Collaborators Nikolaos Apostolakos, Andrea Bonchi, Andrey Belikov, Massimo Brescia, Peter Capak, Jean Coupon, Christophe Dabin, Hubert Degaudenzi, Shantanu Desai, Florian Dubath, Adriano Fontana, Sotiria Fotopoulou, Marco Frailis, Audrey Galametz, Catherine Grenet, John Hoar, Mark Holliman, Ben Hoyle, Olivier Ilbert, Martin Kuemmel, Clotilde Laigle, Giuseppe Longo, Henry Joy McCracken, Martin Melchior, Yannick Mellier, Joe Mohr, Nicolas Morisset, Stéphane Paltani, Roser Pello, Stefano Pilo, Gianluca Polenta, Maurice Poncet, Roberto Saglia, Mara Salvato, Marc Sauvage, Marc Schefer, Marco Scodeggio, Stella Seitz, Santiago Serrano, Marco Soldati, Andrea Tramacere, Rees Williams, Andrea Zacchei, etc. data processing challenges 2

3 Outline 1.Science and mission overview 2.Instruments and data analysis 3.Software development 4.Software integration and operation preparation 5.Swiss Science Data Center (SDC-CH) major tasks This presentation targets a non- audience Focus on data processing aspects of the mission data processing challenges 3

4 An Ever Expanding Universe? Physics Nobel Price 2011 Discovery of the accelerated expansion of the Universe through distant supernovae observations Perlmutter, Schmidt and Riess data processing challenges 4

5 The mission main goal 27% 5% 68% What is the Nature of the Dark Matter and Energy? data processing challenges 5

6 The mission ESA medium scientific Cosmology mission selected in 2011 Soyuz launch from Kourou to L2 in 2020 and 6 year mission Survey of 15'000 square degrees : Optical and NIR images and NIR spectra shape and distance measurements of billions of galaxies Constraints on cosmology models from different types of measurements (or probes): Gravitational (strong and weak) lensing Baryonic Acoustic Oscillation (BAO) Integrated Sachs-Wolfe (ISW) effect (galaxy clusters) Redshift-space distortions (Kaiser effect) data processing challenges 6

7 Weak lensing illustration Masses bend light paths! data processing challenges 7

8 data processing challenges 8

9 3D dark matter map, COSMOS field NASA, ESA, R. Massey (California Institute of Technology). data processing challenges 9

10 Baryonic Acoustic Oscillation data processing challenges 10

11 Strong Lensing Beautiful images... but, only legacy science! will lead to the detection of very large numbers of strong lenses at cluster and galaxy scales data processing challenges 11

12 The Spacecraft 1.2m Korsch Silicon Carbide primary mirror data processing challenges 12

13 data processing challenges 13

14 and ground photometry for PHZ! Dark Energy Survey (DES), Kilo-Degree Survey(KiDS), LSST (?), Javalahambre/Spain, Subaru/Japan (?), CFHT/Canada data processing challenges 14

15 Processing budget Storage (PB) Computing (kilo cores / year) Numbers from Christophe tk1 data processing challenges 15

16 Processing functional break down MOC Ground Station SIM : simulated data OPS SOC Level 1 LE1 VIS NIR SIR Level E EXT SIM Level 2 Level S MER SPE PHZ SHE Level 3 LE3 VIS : visible calibrated frames NIR : near IR calibrated frames SIR : calibrated 1-D spectra EXT : calibrated ground frames MER : catalog with consistent photometry and spectroscopy SPE : spectroscopic redshifts PHZ : photometric redshifts SHE : shape measurements LE3 : high-level processing data processing challenges 16

17 data flow data processing challenges 17

18 SGS organization data processing challenges 18

19 SGS organization OU task : Algorithms specification & validation data processing challenges 19

20 SGS organization SDC task : Software development and Data processing OU task : Algorithms specification & validation data processing challenges 20

21 Software Development C++ and Python languages One reference platform Linux from the Red Hat family (currently CentOS7) Set of common libraries (EDEN) Software development on a virtual machine (LODEEN) RPM packaging XML-based common data model A common building and packaging framework data processing challenges 21

22 Elements framework Elements is a Cmake-based building and packaging framework (capitalizing on CERN expertise) featuring : a standard source code structure easy software building according to CMakeLists.txt instructions automated RPM packaging (make rpm) basic services, such as program option handling and logging data processing challenges 22

23 Projects (Elements Framework) data processing challenges 23

24 Distributed data processing SDC Processing & Local Archive SOC Processing & Local Archive Metadata Data Base SDC Processing & Local Archive 10+ SDCs involved Central metadata database SDC Processing & Local Archive Data centric approach: software runs were the required data has been shipped In each SDC SDC Processing & Local Archive SDC Processing & Local Archive Data Products Metadata Updates Metadata Queries SDC Processing & Local Archive Distributed processing management tools Computing infrastructure for Archive System processing storage data processing challenges 24

25 Distributed Processing Infrastructure Processing Control (Processing Order Definition) SDC y SDC z SDC x Archive System Science Archive Meta-Data Storage XML Data Storage File Infrastructure Abstraction Layer (IAL) Software Continuous Integration and Deployment (CernVM FS) Computing Infrastructure data processing challenges Monitoring (Icinga) 25

26 Infrastructure Abstraction Layer Processing Order Definition Metadata Data Base Polls Processing Orders Fetches inputs from EAS and prepares workspace Ingests outputs into EAS Meta Scheduler SDC Data Storage Work Space Compute Nodes Contains all inputs, outputs, intermediary data for pipeline runs. Pipeline Run Server IAL DRM Queuing System Creates and traverses data fow graph IAL Host Submits and Submission Host, HPC monitors HPC jobs data processing challenges 26

27 Challenge-driven development Iterative development through the planning of a number of incremental integration tests Series of challenges for different aspects of the system weak lensing (Great) infrastructure science photometric redshifts Consolidation of the interfaces (Common Data Model) data processing challenges 27

Infrastructure Challenge 6 Processing Control (COORS) (Processing Orders) SDC y SDC z SDC x Archive System Science Archive Meta-Data Storage XML Data Storage File preliminary versions of (almost) all

28 Infrastructure Challenge 6 Processing Control (COORS) (Processing Orders) SDC y SDC z SDC x Archive System Science Archive Meta-Data Storage XML Data Storage File preliminary versions of (almost) all components involving almost all SDCs! Infrastructure Abstraction Layer (IAL) Software Continuous Integration and Deployment (CernVM FS) Computing Infrastructure data processing challenges Monitoring (Icinga) 28

29 Science challenges 2 and 3 MOC Ground Station OPS SOC Level 1 Science 2 challenge (spring 2016) LE1 Level E VIS NIR SIR EXT SIM Level 2 Level S SPE Level 3 Science 3 challenge (spring 2017) MER PHZ SHE LE3 data processing challenges 29

30 SDC-CH major tasks Develop and provide the Elements building and packaging framework to the collaboration Photometric redshift-related software development Phosphoros : template fitting algorithm implementation PHZ pipeline combining template fitting and machine learning algorithms Strong lens detection Contribution to algorithm exploration (Paraficz et al (Tramacere et al Development of a new (SExtractor) framework in C++ data processing challenges 30

31 Phosphoros challenge results data processing challenges 31

32 SExtractor++ A new modular and extensible SExtractor framework For the astronomical and the communities Long term maintenance and evolution perspectives Modern software design API based on interfaces Single responsibility principles Design patterns BOOST plugin system for adding algorithm steps Collaboration between Emmanuel Bertin and the community data processing challenges 32

33 SExtractor++ status Framework ready Simplified aperture photometry : SExtractor comparison! SExtractor Multi-frame model fitting SExtractor++ data processing challenges 33

34 Conclusions challenges: science goals, hardware development, algorithm determination, software development, etc... Challenge-driven development : best approach for building up software systems through large collaborations? Possible extra benefits for the astronomical community: The Elements building and packaging framework Part of the Infrastructure Abstraction Layer (IAL) Science tools, such as Phosphoros and SExtractor data processing challenges 34

35 Thanks for your attention! data processing challenges 35

BIDS 2016 Santa Cruz de Tenerife

BIDS 2016 Santa Cruz de Tenerife 1 EUCLID: Orchestrating the software development and the scientific data production in a map reduce paradigm Christophe Dabin (CNES) M. Poncet, K. Noddle, M. Holliman,