Progress towards an Asia-Pacific Regional Centre Network. Peter Quinn (ICRAR)

Transcription

1 Progress towards an Asia-Pacific Regional Centre Network Peter Quinn (ICRAR)

2 What observatories do Regional Centre? Regional Centre Regional Centre? 2

3 Data rates and volumes ALMA Science Data Products ~ 1 PB/year LSST Science Data Products 4-5 PB/year SKA SKA1-LOW excluding EoR: 3 Gb/s EoR only: 22 Gb/s SKA1-MID: 9 Gb/s TOTAL: 34 Gb/s = 370 TB/day = 130 PB/yr FULL DISCOVERY ARCHIVE : ~ 700 PB/yr LHC High Luminosity LHC 2023 Science Data Products ~ 400 PB/year 3

4 The LHC decision LHC produced 70 PBytes in its first 3 years (expected 15PB/year) High Luminosity LHC (2023) 400 PB/ year and multi Exabyte data sets for CMS and ATLAS on SKA timescale 2001 LHC realized compute/store cost exceeded accelerator and detector costs Created the Worldwide LHC Computing Grid ( the grid ) separately funded, reviewed and governed but fused with LHC facility/ CERN, MOUs from contributing organisations and consortia Diversify funding source opportunities More completely engage the community 4

5 A change in approach

6 A change in approach

7 The next 5-10 years CHILES x10 VLASS x10 ngvla x10 FAST/ASKAP/MWA x100 x10 SKA 6

8 The next 5-10 years Not CHILES x10 VLASS x10 ngvla possible without the formation of SKA Regional Centres x10 and Alliances FAST/ASKAP/MWA x100 x10 SKA 6

9 SRCCG SKAO Regional Centre Coordination Group (SRCCG): Formed - first three telecons: 23 Sept, 19 Oct, 16 December Members: Antonio Chrysostomou (SKAO, Chair), Severin Gaudet (Canada), Michael Wise (Europe), Jasper Horrell (RSA), Andy Connolly (LSST), Ian Bird (CERN), Miles Deegan (SKAO), Peter Quinn (Aus/NZ), Nick Rees (SKAO), Yogesh Wadadekar (India), Shenghua Yu (China) Terms of Reference: 7

10 SRCCG SKAO Regional Centre Coordination Group (SRCCG): Formed - first three telecons: 23 Sept, 19 Oct, 16 December define a minimum set of requirements for individual SRCs; Members: develop a Antonio set of requirements Chrysostomou for the SRC network (SKAO, as a whole, Chair), including Severin plans Gaudet for the (Canada), physical network Michael connecbng Wise SRCs, (Europe), both together Jasper and with Horrell the SKAO (RSA), sites; Andy Connolly develop (LSST), the requirements Ian Bird for implemenbng (CERN), Miles an SKA data Deegan archive; (SKAO), Peter Quinn idenbfy the requirements for, and coordinate the development of, technical interfaces (Aus/NZ), between Nick the SRCs Rees and the (SKAO), SKA; Yogesh Wadadekar (India), Shenghua Yu (China) dray the MoU to be agreed between the SRCs and the SKA Observatory that accept the Terms stated of minimum Reference: requirements and SKA data access policies; The SKA Regional Centre CoordinaBon Group (SRCCG) is instructed to: develop a set of data challenges to test and verify the capabilibes and resources of each SRC, as well as the SRC network; develop an accreditabon process for the SRCs, including branding and appearance policies; develop a process to ensure that soyware tools are interoperable across the SRC network; develop a policy for the inclusion of user-generated advanced data products into the SRC archive; develop SKA data heritage and access policies to that heritage data set for approval by the SKAO; and invesbgate models for the future governance of the collaborabve network of SRCs. 7

11 SRCCG SKAO Regional Centre Coordination Group (SRCCG): Formed - first three telecons: 23 Sept, 19 Oct, 16 December Members: Antonio Chrysostomou (SKAO, Chair), Severin Gaudet (Canada), Michael Wise (Europe), Jasper Horrell (RSA), Andy Connolly (LSST), Ian Bird (CERN), Miles Deegan (SKAO), Peter Quinn (Aus/NZ), Nick Rees (SKAO), Yogesh Wadadekar (India), Shenghua Yu (China) Terms of Reference: Typical Data and Processing Aggregate Requirements: PB/year of data products into a persistent, interoperable and curated global science archive for SKA PF processing Big Known Unknowns: SKA operational boundaries and interfaces (e.g. User Support) SKA KSP team work flows SRC Funding, SRC Alliance, SRC operations, policies and costs 7

12 SRCCG Progress SKA REGIONAL CENTRES: BACKGROUND AND FRAMEWORK Document number... SKA-TEL-SKO Document Type... RSP Revision Author... A. Chrysostomou, SRCCG Date Document Classification... UNRESTRICTED Status... Pending Approval 8

13 SRCCG Progress SKA REGIONAL CENTRES: BACKGROUND AND FRAMEWORK Document number... SKA-TEL-SKO Document Type... RSP Figure 1: A Revision network... of SKA Regional Centres receives science data products from the SKA Observatory. 01 The SRCs provide Author users... with access to SKA science data products, as well as A. the Chrysostomou, tools and processing SRCCG power necessary Date to fully... exploit the science potential of those products, irrespective of their geographical location Document Classification or whether... their local region or country hosts an SRC. UNRESTRICTED Status... Pending Approval 8

14 SRCCG Progress SKA Observatory SRCs Users Phase 1: Proposals Phase 2: Project Design Planning & Observa9on Data Processing (inc. QA) SKA REGIONAL CENTRES: BACKGROUND AND Long Term FRAMEWORK Preserva9on Document number... SKA-TEL-SKO Delivery Document Type... RSP Figure 1: A Revision network... of SKA Regional Centres receives science data products from the SKA Observatory. 01 The Author... Science SRCs provide users with access to SKA science data products, as well as A. the Chrysostomou, tools and processing SRCCG power necessary Date to fully... exploit the science potential of those products, irrespective of their geographical location Document Classification or whether... their local region or country hosts an SRC. UNRESTRICTED Figure 2: Process flow and responsibility diagram for the operational workflow of the SKA Observatory, the SRCs Status... Pending Approval and the user community. Horizontal arrows indicate interactions and vertical lines depict the range of involvement. Open circles indicate processes solely undertaken by the SKA Observatory. 8

15 ASRCWG Australian and New Zealand SKA Coordination Committee (ANZSCC) SKA Regional Centre Working Group (ASRCWG): Formed - first two meetings 9 August and 12 December Members: Chair: Peter Quinn (ICRAR), ASKAO: David Luchetti, CSIRO: Douglas Bock/David McConnell, SKA Operations: Sarah Pearce, ICRAR: Andreas Wicenec, CAASTRO: Naomi McClure-Griffiths, AAL: Yeshe Fenner, AARNET: Peter Elford, Pawsey Centre: Neil Stringfellow, ANZSCC SAC: Carole Jackson, ACAMAR: Lister Staveley-Smith, ASKAIC: Geof Heydon, New Zealand: Willem van Straten, Melanie Hohnston-Hollitt Terms of Reference: 9

16 ASRCWG Australian and New Zealand SKA Coordination Committee (ANZSCC) 1. SKA To Regional define the Centre Australian Working SKA Regional Group (ASRCWG): Centre scope, opportunities, requirements Formed - first and two potential meetings funding 9 August opportunities, and 12 December in line with SKAO Operational Members: Chair: planning Peter and Quinn the SKAO (ICRAR), Board s ASKAO: response David to Luchetti, the CSIRO: recommendations Douglas Bock/David of the McConnell, SKA Data SKA Flow Operations: Advisory Committee Sarah Pearce, 2. ICRAR: To explore Andreas opportunities Wicenec, CAASTRO: to expand the Naomi Australian McClure-Griffiths, SKA Regional AAL: Centre Yeshe Fenner, activities AARNET: to include Peter collaboration Elford, Pawsey with similar Centre: activities Neil Stringfellow, China, New ANZSCC Zealand SAC: and Carole the broader Jackson, Asia-Pacific ACAMAR: Region Lister Staveley-Smith, with a particular focus ASKAIC: on business Geof Heydon, case development New Zealand: and Willem precursor van Straten, enabled Melanie technological Hohnston-Hollitt and scientific programs 3. Terms To initiate of Reference: a detailed study (ERIDANUS) of the data and processing requirements and costs within an Australian SKA Regional Centre in support of Australian Survey Science with the SKA precursors and SKA-1 9

17 China-Australia strengths SCIENCE : ACAMAR (Australian-ChinA ConsortiuM for Astrophysical Research) alliance - optical/ir, radio, theoretical and Antarctic astronomy Future: FAST/ASKAP/MWA surveys, VLBI, Kunlun TECHNOLOGY: CSIRO Multi-beam technologies for FAST Low frequency technology - 21CMA, MWA+ Antennae design DATA: Joint effort on SKA SDP systems SDP prototypes - FAST, MWA and ASKAP NGAS data flow systems for FAST Developing regional VO capabilities April 2016 Capabilities and opportunities to develop regional solutions for SKAclass data flows and processing 10

18 ICRAR - SHAO May

19 Data Canals and Rivers FUNDED: 15 July Exascale Research Infrastructure for Data in Asian-Pacific astrononomy Using the SKA Advanced European Network of E-infrastructures for Astronomy with the SKA Launched April eridanus.net.au 12

20 ERIDANUS The ERIDANUS Project: A three year, bottom-up, design study commencing in 2017, aimed at deploying prototype data intensive research infrastructure and middleware, between and within Australia and China, capable of addressing SKA-class data and processing challenges. Will respond to challenges identified by the SRCCG and will work in a coordinated and collaborative manner with the European Aeneas Project. SKA-1 Scale Technology Development Projects Precursor Science Projects 13

21 ERIDANUS The ERIDANUS Project will engage with: The astronomical communities within Australia and China Industrial partners The providers of networking and computing research infrastructure National and international committees coordinating efforts on SKA regional centre developments SKA-1 Scale Technology Development Projects Precursor Science Projects 14

22 ERIDANUS The ERIDANUS Project: Founded by ICRAR and SHAO in November now includes CSIRO and interest from CADC - Focus: Data flow architectures SKA-1 Scale Technology Development Projects Execution frameworks Strategies and systems to monitor and control cost Networking optimisation Cloud computing options OPEN INVITATION FOR ALL TO JOIN! Precursor Science Projects 15

23 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

24 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

25 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

26 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

27 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

28 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

29 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

30 Australia-China SKA Big Data Meeting April , Shanghai Zhangjiang High Technology Park Funded by DIIS and MOST 50 participants from Aus and China with observers from SKAO and CADC Networking, Near Real-time (Near real-time, (non-)imaging algorithms and software components), Processing frameworks and software systems, Collaborative data reduction support software, Data management software systems for globally distributed data, Data Mining technologies, Hardware platforms, Global astronomical data dissemination at SKA scale and VO software and technologies 16

31 2017/18 ERIDANUS deployment of processing and storage prototypes and scaling tests to SKA-1 scale using largest facilities available in Australia and China Focused on precursor needs: ASKAP, MWA and FAST Focused on large-scale surveys: CHILES+, GLASS, IMAGINE, as workflow and CLOUD testbeds Focused on ASKAPsoft, DALiuGE and codeveloped technology with industry: e.g. Nyriad (NZ) on new processing/storage HW (GPU optimisation) 2nd Aus-China Big Data Workshop ~ Nov/Dec 17 Australian SKA Regional Centre requirements definition (with SRCCG and KSPs) and Australian government funding application (2017?) 17

32 2020/21 Funded Regional Centre initiatives in China and Australia ASKAP, MWA and FAST in full science operations using components from ERIDANUS developments First science results from Australia/China precursor science teams using ERIDANUS infrastructure SKA Regional Centre Alliance formed First ERIDANUS and AENEAS success in SKA data challenges SKA-1 construction started Joint Australian and China (with others) Asia/Pacific Regional Centre network proposal to governments for an SKA-1 class capability by

33 SKA in Australia SKAO SKA Observatory RSA AUS SKA Community AUS Node 1 Node 2 Node 3 MRO Pawsey Geraldton SKA Operations Perth SKA Regional Centre Node 2 Node 1 Scheduling Execution QC Calibration Pipelines Data Packages Site Maintenance Community Support Survey Science Team Support High level data products Science, Engineering, DIA innovation Education/Outreach + Community 19

34 Asian Pacific SKA Regional Centre Network SKAO SKA Observatory RSA AUS SKA Community AUS China SKA Regional Centre Community Support Survey Science Teams High level data products Science, Engineering, DIA, innovation Education/Outreach Pathfinder Science Data Technology Regional opportunities Innovation SKA Regional Centre Community Support Survey Science Teams High level data products Science, Engineering, DIA, innovation Education/Outreach Node Node Node Node Node Node 20