Designing the Future Data Management Environment for [Radio] Astronomy. JJ Kavelaars Canadian Astronomy Data Centre

Size: px

Start display at page:

Download "Designing the Future Data Management Environment for [Radio] Astronomy. JJ Kavelaars Canadian Astronomy Data Centre"

Preston Price
6 years ago
Views:

1 Designing the Future Data Management Environment for [Radio] Astronomy JJ Kavelaars Canadian Astronomy Data Centre

2 2 Started working in Radio Data Archiving as Graduate student at Queen s in 1993

3 Canadian Astronomy Data Centre (CADC) Data archive housing collections from multiple (+10) telescopes: Currently house about 2 PB of telescope archive data Development group working on standards in data archive system via the International Virtual Observatory Alliance (IVOA) including: Common Archive Observation Model (CAOM and ObsCore) VOSpace - Open storage system protocol. SIAP - Simple Image Access Protocol Database Table Access Protocol (TAP) + Astronomy Data Query Language Group Management Service Authentication and Access Lead development and support for the Canadian Advanced Network For Astronomical Research (CANFAR) - Cloud computing in Astronomy Research Astronomers investigating Dark Energy, Quasars, Galaxy Evolution, Stellar Atmosphere, the Trans Neptunian Region and Machine Learning in image and spectroscopy classification. No one doing Radio astronomy.. but we are branching out. 3

4 Example IVOA Standard used at MWA siap.xml -- a standard SIAP interface as defined by the IVOA to access collections of celestial images; SIAP clients use gleam-vo.icrar.org/gleam_postage/q/siap.xml? to access the service VOTable Output Fields 4

5 JCMT Data Delivery Location since

Many research projects growing in the volume and complexity of data analysis required.

are growing in size and becoming increasingly geographically distributed Visualization of

6 The current data problem for our users. Multiple PBs of data stored in archives and research repositories. Many research projects growing in the volume and complexity of data analysis required. Entire CADC archive delivered to users annually, over the internet Research partnerships are growing in size and becoming increasingly geographically distributed Visualization of complex datasets not obvious or straight forward. Resource requirements not always available near the expert. Data transfer overheads often exceed computation duration. 6

7 7

8 The data centre landscape CADC operates an archive platform. Platform is used by multiple facilities. System architecture is facility (and energy) agnostic Hardware costs shared: NRC, Compute Canada and CANARIE CANFAR : science collaboration and data processing/storage platform operated by CADC Platform attempts to make generic access to storage and computing straightforward Acts as a focus point for Observational Astronomy resources allocation on Compute Canada Similar platforms exist / emerging in other disciplines CFI / CANARIE / NRC fund(ed) the development 8

9 CANFAR : Cloud storage and processing for astronomy. The computing and data are in the same computing centre. Keep the data local to the compute. The cloud abstracts your access to the data centre. When you run your computing on the cloud the jobs are moved to the computing centre that holds the data or the computing requests the data from the local storage. Should be transparent to users. CADC/CANFAR is currently distributed across three sites in British Columbia. About 1000 cores in a Cloud/VM system with about 4 PB of User and Archive storage. 9

10 10

11 NRC CC CC CADC - CANFAR 11

12 DC2 Federation Layer SC3 DC1 12

13 CFI Innovation Fund What CADC/CANFAR will provide (straight from proposal). Long-term archiving and service: We require enhanced access to and longterm archiving of Science Ready Data Products (SRDPs). This item, to be provided by CADC, includes: 1. Indefinite access to the SRDPs, 2. Customizations to the CANFAR cloud computing platform to meet project requirements for running the Processing Software Stack. 3. Custom design for a query interface for SRDPs in the CADC data portal, and 4. Long-term maintenance of interfaces and data formats to preserve the results of the project indefinitely 13

14 CFI Cyber Infrastructure Fund - PROPOSAL Improved systems for CANFAR users, including: New interactive data processing environments Database systems that support large astronomical catalogs provided by users / science teams Data processing work flows for astronomy Improved batch processing and data system Data Exploration environment. Cross-correlation Developed and design in coordination with specific projects but made general enough for wide use in astronomy. 14

15 Thank you JJ Kavelaars Canadian Astronomy Data Centre Tel:

Data Centres in the Virtual Observatory Age

Data Centres in the Virtual Observatory Age David Schade Canadian Astronomy Data Centre A few things I ve learned in the past two days There exist serious efforts at Long-Term Data Preservation Alliance