Data Processing for the Square Kilometre Array Telescope

Transcription

1 Data Processing for the Square Kilometre Array Telescope Streaming Workshop Indianapolis th October Bojan Nikolic Astrophysics Group, Cavendish Lab University of Cambridge Project Engineer & Architecture Task Leader SKA Science Data Processor Consortium

2 Not covered in this talk Not covered: Science Objectives of the SKA Timelines, project programmatic and organisational structure Design of the receptors (aperture arrays & dishes) and their associated digital electronics/communications equipment Also quite a few spare slides on topics covered. Find me later if you d like information on these

3 THE SQUARE KILOMETRE ARRAY TELESCOPE

4 What is the Square Kilometre Array (SKA)?

5 Mid frequency dishes and dense aperture array concept

6 Low frequency array and the survey telescope concept

7 Phased Aperture array for 0 MHz

8 Phased Aperture array: million antennas

9 Scientific Context a partner to ALMA, EELT, JWST Credit:A. Marinkovic/XCam/ALMA(ESO/NAOJ/NRAO) Credit:ESO/L. Calçada (artists impression) Credit: Northrop Grumman (artists impression) Credit: SKA Organisation (artists impression)

10 Scientific Context a partner to ALMA, EELT, JWST ALMA: high precision sub-mm antennas Completed in Budget ~. bn USD European ELT ~m optical telescope Completion ~ Budget ~. bn EUR Credit:A. Marinkovic/XCam/ALMA(ESO/NAOJ/NRAO) JWST:.m space near-infrared telescope Launch Budget ~ bn USD Credit:ESO/L. Calçada (artists impression) Square Kilometre Array Two next generation low-frequency arrays Completion ~ for Phase Budget 0. bn EUR for Phase Construction Credit: Northrop Grumman (artists impression) Credit: SKA Organisation (artists impression)

11 What will the Square Kilometre Array (SKA) be? Radio Telescope Currently in Design Major Engineering Project Major ICT Project Makes Images of the Sky at radio (m-cm) wavelengths ~0 more sensitive than current telescopes Complements ALMA, JWST (successor to Hubble), and E-ELT Construction begins Full operations expected at end Significant funds already committed by participating countries Two remote desert sites >0k receiving elements Subject of this talk!

12 SCIENCE DATA PROCESSOR CONTEXT

13 The SKA Observatory Phase : SKA People, buildings, roads, ground works, communications, electrical power, maintenance, transportation, catering at desert sites. Data Processing: general computing Digital Signal Processing: FPGA + (maybe) ASICS & GPUs Receptors: Aperture Arrays and Dishes

14 SKA Context Diagram Monitor and Control SKA Low: Low Frequency Aperture Array LFAA Correlator/ Beam Former Science Data Processor Implementation (Australia) Pulsar Search Processor (Australia) SKA Mid: Dish Antennas with Single-Pixel feeds SKA Mid Correlator/ Beam Former Pulsar Search Processor (South Africa) Science Data Processor Implementation (South Africa) Astronomers These are offsite! (In Perth & Cape Town)

15 Role of computing/processing in SKA: COMPUTING IS THE MAJOR PART OF SKA TELESCOPES BY DESIGN

16 Large D vs Large N GBT 0-m diameter telescope SKA LFAA prototype array No aim: collect as many photons as possible No aim: maximum separation of collectors -> achieve high angular resolution

17 Basics of Interferometry how signals are combined Astronomical signal (EM wave) s B. s Visibility: Detect & amplify Digitise & delay Correlate Integrate Process ( == SDP) B X X X X X X Calibrate, grid, FFT V(B) = E E * = I(s) exp( i w B.s/c ) Resolution determined by maximum baseline q max ~ l / B max Field of View (FoV) determined by the size of each dish q dish ~ l / D SKY Image

18 SKA SCIENCE DATA PROCESSOR OVERVIEW

19 Key Characteristics of SKA Data Processing Noisy Data Sparse and Incomplete Sampling Corrupted Measurements Multiple dimensions of data parallelism Very large data volumes, all data are processed in each observation Corrected for by deconvolution using iterative algorithms (~ iterations) Corrected by jointly solving for the sky brightness distribution and for the slowly changing corruption effects using iterative algorithms Loosely coupled tasks, large degree of parallelism is inherently available

20 SDP Top-level Components & Key Performance Requirements -- SKA Phase Science Data Processor Telescope Manager SDP Local Monitor & Control C S P Tera Byte/s Data Processor High Performance ~0 PetaFLOPS Data Intensive ~0 PetaBytes/observation (job) Partially real-time ~s response time Partially iterative ~ iterations/job (~ hour) Long Term Archive High Volume & High Growth Rate ~0 PetaByte/year Infrequent Access ~few times/year max Delivery System Data Distribution ~0 PetaByte/year from Cape Town & Perth to rest of World Data Discovery Visualisation of 0k by 0k by 0k voxel cubes Regional Centres & Astronomers

21 The challenges? Power efficiency Funding agencies more tolerant of cap-ex then power op-ex Scalability of software Hardware roadmaps indicate h/w will reach requirements Demonstrated software scaling is only /00 th of requirement Project risks Inevitable significant interaction between software engineers and idiosyncratic domain specific knowledge Software project Extensibility, system scalability, maintainability SKA is the first milestone expecting significant expansion in the s 0yr observatory lifetime

22 Relevance of Streaming technologies Multiple Loosely Coupled Task Yes Within each iteration & between observation BSP Yes: a few (~) iterations Synchronisation at end of each iteration Relatively small synchronised state (sky model + calibration) Workflow yes Streaming yes (but not interactive) Steering yes (feed back of calibration solutions to the receptors) Data Driven yes & yes In the dataflow execution model sense (i.e., not demand-driven) In the programming model sense (i.e., a declarative description of how to reduce the data are attached to the data themselves)

23 Why streaming? Detecting transient phenomena Need to generate alerts latest about ~s after receiving the data Impractical to store data for later processing Data rate in ~ TeraByte/s -> stream but with data chunked by observation

24 SKA Science Data Processor ARCHITECTURE HIGHLIGHTS

25 Programming model Hybrid programming model: Dataflow at coarse-grained level: About million tasks/s max over the whole processor (-> ~s 0s milli second tasks), consuming ~0 MegaByte each Static scheduling at coarsest-level (down to data-island ) Static partitioning of the large-volume input data Dynamic scheduling within data island: Failure recovery, dynamic load-balancing Data driven (all data will be used) Shared memory model at fine-grained level e.g.: threads/openmp/simt-like ~0s active threads per shared memory space Allows manageable working memory size, computational efficiency

26 Why? Shared memory model essential at finegrain to control working memory requirements Dataflow (but all of these are still to be proven in our application): Load-balancing Minimisation of data movement Handling failure Adaptability to different system architectures

27 Fault Tolerance Non-Precious Data Classify arcs in the dataflow graph as precious or non-precious Precious data are treated in usual way failover, restart, RAID, etc. Non-precious data can be dropped: If they are input to a map-type operation then no output If they are input to a reduction then result is computed without them Stragglers outputting non-precious data can be terminated after a relatively short time-out

28 The non-precious data concept - illustration FULL DATASET FULL DATASET SPLIT BY FREQ SPLIT BY FREQ FREQ FREQ FREQ FREQ NF FREQ FREQ FREQ FREQ NF GRID GRID GRID GRID GRID GRID GRID GRID FFT FFT FFT FFT FFT FFT FFT FFT Single Frequency Image Single Frequency Image Single Frequency Image Single Frequency Image Single Frequency Image Single Frequency Image Single Frequency Image Single Frequency Image Reduce by pixelvise addition Freq Image Freq Image Freq Image Freq Image Reduce by pixelvise addition Reduce by pixelvise addition Combined Image Combined Image Normalise by Normalise by Combined Image Combined Image

29 Data Ingress and Interconnect Concept CSP Node CSP Node CSP Node CSP Node CSP Node CSP Node CSP Node CSP Node CSP Node INGEST LAYER st stage BDN st stage LLN st stage BDN st stage LLN st stage BDN nd stage BDN nd stage LLN nd stage BDN nd stage LLN nd stage BDN nd stage LLN nd stage BDN nd stage LLN nd stage BDN rd stage switch Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Compute Island Science Archive switch Science Archive switch High Performance Buffer Medium Performance Buffer Storage Pod Storage Pod Archive Network Low-Latency Network Long Term Storage DELIV Bulk Data Network

30 Final Remarks From SKA-SDP Phase perspective: Probably all of the paradigms and ideas we need are available Most technologies required for implementing these are available but scattered between many products and platforms Need to move towards integrating these together and stable APIs

31