Large Scale Data Management of Astronomical Surveys with AstroSpark

Transcription

1 Large Scale Data Management of Astronomical Surveys with AstroSpark Mariem BRAHEM 1,2, Karine ZEITOUNI 1, Laurent YEH 1 (1) DAVID Lab University of Versailles (2) CNES Centre National d Etudes Spatiale - Toulouse XLDB th Extremely Large Databases Conference Clermont-Ferrand, October 2017

2 Context & Motivation Big Data in the field of Cosmology 3D map of our Galaxy 1 billion stars observed >1PB Dec >10 billions of objects > 15PB for the catalog > 2020 Gaia mission LSST Project Ø Astronomers need an efficient solution for large scale astronomical data handling. AstroSpark - XLDB

3 Contributions We propose new algorithms and a framework AstroSpark Ø which extends Apache Spark, a distributed in-memory computing engine to process and analyze astronomical data ² Implements operators such as Cone-Search, Cross-Match, knn, Ø offers an expressive programming interface by supporting the unified query language ADQL Ø combines data partitioning and indexing with HEALPIX, a pixelization of data on the sphere, to speed up query processing. ² e.g., HX-Match uses HEALPIX to speed-up Cross-Match Ø implements a query optimizer and provides a set of customized strategies for astronomical queries. AstroSpark - XLDB

4 AstroSpark Architecture Input Data Querying system Astronomical Data Query Language (ADQL) SELECT * FROM gaia JOIN tycho2 ON 1=CONTAINS ( POINT( ICRS, gaia.ra, gaia.dec), CIRCLE( ICRS, tycho2.ra, tycho2.dec, 2/3600)) Data Partitioning Query Language (ADQL) Query Parser Query Optimizer (extended Catalyst) Int. Virtual Observatory Alliance IVOA Healpix library Storage (HDFS) SPARK Core AstroSpark - XLDB

5 Healpix Based Data Par00oning ü Locality: Close data points are likely to be in the same partition & all points withing a Healpix pixel belong to the same partition ü Balance: Partitions have roughly the same size and adapt to data density ü Each node can process many partitions Range = Range = Range = Node 1 Node 2 Node 3 AstroSpark - XLDB 2017 Visualization under ALADIN A tool provided by 5

6 Cross-Matching Example Iden0fy and correlate objects belonging to different observa0ons Could be expressed in Spark SQL : SELECT * FROM R JOIN S ON (2*ASIN(SQRT(SIN((DEC2 - DEC)/2) * SIN((DEC2 - DEC)/2) + COS(DEC2) * COS(DEC) * SIN((RA2 - RA)/ 2) * SIN((RA2 RA)/2))) <= ɛ) But untractable: Cross-matching only 200,000 records of Gaia and Tycho-2 takes 13,6 hours, and more than 12 days for 5 million objects in Gaia and tycho-2! ɛ 6 R S

7 Cross-Matching in AstroSpark Ø HX-Match leverages the space indexing & partitioning along with the HEALPIX NASA Library to guide the data access and limit the pairwise distance computation. Ø Substitutes the costly cartesian product by an Equi-join + Filter Query Plan in Spark SQL Query Plan in AstroSpark 7

8 Results of cross-matching: GAIA DR1 TYCHO2 Logarithmic scale Logarithmic scale AstroSpark - XLDB Gain of the partition materialization

9 Cone Search & knn Search Evalua0on on GAIA DR1 Effect of varying data size on Cone Search 9

10 First Results & Impacts Validated for 3 operators & ADQL queries on real datasets Experiments have shown that AstroSpark is effective in processing astronomical data, scalable and overperforms the state-of-the-art solutions. Publications: Ø Ø Ø Ø M. Brahem, K. Zeitouni & L. Yeh, HX-MATCH: In-Memory Cross- Matching Algorithm for Astronomical Big Data, International Symposium on Spatial and Temporal Databases (SSTD 2017). M. Brahem, K. Zeitouni & L. Yeh, Large Scale Data Management of Astronomical Surveys with AstroSpark, Conference on Big Data from Space (BiDS 2017) K. Zeitouni, M. Brahem & L. Yeh, Large Scale Data Management of Astronomical Surveys with AstroSpark, European Week of Astronomy and Space Science (EWASS 2017) M. Brahem, K. Zeitouni & L. Yeh, AstroSpark: towards a distributed data server for big data in astronomy. Proceedings of the 3 rd ACM SIGSPATIAL PhD Symposium. ACM 2016 AstroSpark - XLDB