PROCESSING THE GAIA DATA IN CNES: THE GREAT ADVENTURE INTO HADOOP WORLD

Transcription

1 CHAOUL Laurence, VALETTE Véronique CNES, Toulouse PROCESSING THE GAIA DATA IN CNES: THE GREAT ADVENTURE INTO HADOOP WORLD BIDS 16, March 15-17th 2016

2 THE GAIA MISSION AND DPAC ARCHITECTURE AGENDA THE DPCC ARCHITECTURE FIRST DPCC RESULTS FIRST LESSONS LEARNED ON HADOOP 2

3 THE GAIA MISSION AND DPAC ARCHITECTURE AGENDA THE DPCC ARCHITECTURE FIRST DPCC RESULTS FIRST LESSONS LEARNED ON HADOOP 3

4 The Gaia Mission An ESA mission to build a 3D map of 1 billion stars from our Galaxy Gaia launched on December 19th 2013 by a Soyouz-Fregat from Kourou The 2 ton satellite is orbiting around Lagrange L2 point, 1.5 million kilometres from Earth The mission is foreseen to last at least 5 years Scientific data processing delegated to the DPAC Consortium 4

5 The Gaia data flow ~30 GB/day 8hrs/day ESOC MOC Cebreros / New Norcia ESAC SOC 5 Credit: DPACE

6 DPC various architectures overview Postgres-XC 6 Credit : DPACE

7 AGENDA THE GAIA MISSION, DPAC CHALLENGES AND ARCHITECTURE THE DPCC ARCHITECTURE FIRST DPCC RESULTS FIRST LESSONS LEARNED ON HADOOP 7

8 DPCC Architecture DPCC challenges to fulfill 1 billion stars to process, each star seen in average 80 times Some tables up to 80 billions rows High Level of parallelization with several chains running at the same time Various kinds of complex algorithms (object by object or global processings) Increasing volume all along the mission, up to 3PB Hadoop solution selected in 2010 Horizontal linear scalability allowing storage and computing power growth Parallelization with process localization Map Reduce and Resource Manager (YARN) : highly distributed processing HDFS: distributed data storage PHOEBUS to orchestrate all the processings in the cluster 8 8

9 The DPCC platforms Based on standard servers Dell PowerEdge *Intel E CPU with 6 cores@2,5ghz» 64 GB RAM (5.33 GB/core)» 3*4TB 7.2kRPM disks Operational Platform 2*Intel E CPU with 10 cores@2,5ghz 128 GB RAM (6.4 GB/core) 3*4TB 7.2kRPM disks Servers bought progressively, according to computing and storage needs 72 Calculus nodes (1152 cores/240 TB HDFS/20TB GFS) today 400 Datanodes (~6000 cores) at the end of the mission Validation platform Same architecture as the OPS one 28 Datanodes 512 cores 100 TB effective HDFS 8 TB GFS 9 BIDS Tenerife - March 15th-17th 2016

10 GaiaWeb: Data access for the scientific community Web portal enabling the scientific community to access data produced and stored on DPCC clusters Hard-wired statistical plots Processing chains behavior / status follow-up Several million of indexes into an ElasticSearch engine: real-time/reactive access On-demand queries: To extract data for validation purposes / problems investigation SQL-like queries translated into jobs submitted on the cluster, executed as Map/Reduce tasks 10

11 AGENDA THE GAIA MISSION, DPAC CHALLENGES AND ARCHITECTURE THE DPCC ARCHITECTURE FIRST DPCC RESULTS FIRST LESSONS LEARNED ON HADOOP 11

12 First DPCC results using Hadoop (1/2) Daily chains executed on the 1152 cores of the OPS platform : Spectroscopic Daily chain in routine mode Executed everyday as soon as data are received from DPCE Around 10 millions observations processed in about 6hours Solar System Objects Daily chain Not yet run in routine mode (still need some scientific improvements) But the chain manages to process about 64 Millions of observations (with solar objects) on 2 hours Both chains can run in parallel, with a good distribution of the jobs in all the available cores. => The relevance of the Hadoop solution is proven 12

13 First DPCC results using Hadoop (2/2) Cyclic chains executed on the VAL platform : CU4 NSS CU8 Apsis Total Duration # Objects Duration (ms/objects) Total Duration # Objects Duration (ms/objects) Insertion 03:08: ,002 01:06: ,003 Ingestion 14:00: ,042 06:40: ,018 Processing 02:00: ,037 00:07: ,091 10:10: ,304 Performances strongly dependent on the kind of processings Insertion in HDFS of input data : highly distributed Ingestion : transformation of input data into objects specific to the chain» A lot of joins between different tables, so every data has to be read at least once» Can include some scientific computations, so various performances results Processing» Linked to the scientific algorithms themselves 13

14 AGENDA THE GAIA MISSION, DPAC CHALLENGES AND ARCHITECTURE THE DPCC ARCHITECTURE FIRST DPCC RESULTS FIRST LESSONS LEARNED ON HADOOP 14

15 Performances (1/3) Performances monitoring Performances follow-up very complex A lot of statistics given by Hadoop API for each job, the chain performances view shall be consolidated separately Difficult to extrapolate the chain performances, as it is dependent on the other chains running in parallel some tools are being developed in DPCC to aggregate all these statistics and to make a close-monitoring of performances of each chain To anticipate possible overflow To size the next purchase of hardware 15

16 Performances (2/3) Performances closely linked to the design of the chain Steps with different data access typology (steps that need all the data or filters applied) Consequence : Some steps are fully scalable, others not. 16

17 Performances (3/3) Performances closely linked to the design of the chain 17 Each step of the chain is designed to be executed star by star => highly scalable design The design of the chain shall be scalable to fully benefit from Hadoop mechanisms Need to be anticipated at the beginning of development

18 Hadoop fine tuning (1/3) Quite complex fine tuning to obtain optimised performances Configuration of Hadoop queues Allocate an Hadoop queue for each chain to ensure a given fraction of the cluster capacity to this chain Need to find a good configuration to avoid reserving too many resources to a chain 18

19 Hadoop fine tuning (2/3) Quite complex fine tuning to obtain optimised performances Configuration of Hadoop queues The queue elasticity option A given chain is authorized to use the resources of another queue if not used Good usage of all the available resources Dynamic allocation of resources, so very difficult to analyse the performances of the chain 19 The pre-emption option A job can kill another job that would have overflowed in its queue The priority defined by the queues are respected But a job that is running since hours can be killed whereas it is almost done

20 Hadoop fine tuning (3/3) Storage management / replication tuning Replication 3 is recommended But the replication can be configured for each data written in HDFS => ability to tune the replication according to the criticality of the produced data and the available storage 20

21 Hardware management Very good management of heterogeneous machines Servers of different generations, with different characteristics (number of nodes, memory, disk space) Transparent to DPCC operations A lot of available tools to ease the deployment Complete server lifecycle (from out of the box to the production) Same management and monitoring for 10 or 500 servers 21

22 Conclusions / Perspectives Promising results, but further/deeper performances monitoring still needed To allow Hadoop fine-tuning and to consolidate hardware selection for next purchase The next steps : GDR1 (Aug 2016) : Positions (a, d) and G-magnitudes, at least 90 % of the sky can be covered (objects with single star behavior) July 14 Aug 15 May 16 Dec 16 Dec 17 Cycle 0 13 months Cycle 1 8 months We are here Cycle 2 7 months Cycle 3 12 months Daily chains: CU4/SSO-ST & CU6 Daily GDR2 (Mid-2017) : Five parameter astrometric solution of objects with single star behavior will be released (90% of the sky), Integrated photometry BP/RP, mean radial velocities for objects showing no radial velocity variation GDR3 (Jan 2021): Final catalogue release 2019/2020 End of mission Reprocessing cycles CU6 Global R2 CU8 Apsis CU4 SSO-LTa CU4 NSS CU4 SSO-LTa CU4 NSS CU6 Global R2 CU8 Apsis CU4 SSO-LTb CU4 NSS 22 BIDS 2016 Tenerife - CU4 March EO 15th-17th 2016 CU4 EO CU4 EO

23 Questions?