Computational issues for HI

Transcription

1 Computational issues for HI Tim Cornwell, Square Kilometre Array

2 How SKA processes data Science Data Processing system is part of the telescope Only one system per telescope Data flow so large that dedicated facility is needed Telescope becomes adaptive to e.g. cancel calibration effects Steps are: acquire, edit, calibrate, transform, analyse, with iterative cycles Too much data to allow guiding by humans But analysis step requires some human guidance and performance Analysis rich in visualization, feature identification, catalog queries

3 SKA data processing rates SKA phase 2 SKA phase 1 ASKAP Pawsey Centre tests NCI NF tests NCI Altix tests ASKAP dev cluster Note that Flops numbers are not achieved - we actually get much lower efficiency because of memory bandwidth - so scaling is relative

4 SKA1 data products and transformations Low frequency aperture array Dish arrays 1 ADC outputs SKA 0.3 to 3 TB/s Level Beam-former output Correlator output Visibility data Calibrated data, images and catalogues Validated science data products (released by Science Teams) Enhanced data products e.g. Source identification and association Definition SKA SKA SKA SKA ST ST Responsibility TB/s ~ 100 PB data set read multiple times over several days e.g. 1 year Redshifted Hydrogen survey ~ 4EB

5 Basic questions Do we have the algorithms? Do they scale? Can we afford the computers required?

6 SKA data processing challenges Wide field imaging ASKAP, MWA, LOFAR Imaging with aperture arrays LOFAR, MWA Imaging with phased array feeds ASKAP Imaging with wide bandwidth JVLA, ASKAP Calibration and correction of direction dependent effects LOFAR, MWA,, VLA, MeerKAT, ASKAP All organizations represented at SKA CALIM meetings Challenges being addressed across community High performance computing lags

7 Emerging consensus Major-Minor cycle algorithms Multiple passes necessary AW Projection for wide field of view Various optimisations for processing W snapshots W sort Clean for deconvolution Peeling for bright out-of-field sources

8 Accuracy of deconvolution Both for compact and extended structures Shape of sources Noise floor CLEAN has served us well! THINGS Multi-scale CLEAN helps but is too difficult to control Plenty of evidence that Compressive Sampling does better See e.g. papers by Wiaux and collaborators Need to see CS working on SKA size images Scaling Actual improvements on realistic fields Needed for accurate continuum modeling

9 Basic questions Do we have the algorithms? Do they scale? Can we afford the computers required?

10 Scaling ASKAP Processing to ~10,000 cores, SKA must go to ~10,000,000 cores

11 Basic questions Do we have the algorithms? Do they scale? Can we afford the computers required?

12 Costing calibration and imaging Don t try to count flops, IO Scale existing software to SKA Use small benchmark programs to get info for scaling Gridding, cleaning, reprojection, FFT Scale known configuration (and cost)

13 Calibration and imaging cost model Five pipelines Ingest, Calibration, Continuum, Spectral line, Transients Steps in processing Gridding and degridding visibility data Multi-Scale Multi-Frequency CLEAN Resources Processing Memory Fast storage 2 T clean = µ clean N scales N Taylor N iterations N psf

14 Gridding/degridding model w T ws = µ grid N vis ρ 2 rms w max 2 N +µ pixels h obs reproj ρ 2 R 2 2 T F + R A + µ FFT 2ρR obs F T A + N int N 2 pixels log 2 2 ( N pixels ) AW snapshots algorithm Identify best-fit plane in uvw space Use AW Projection to move points onto this plane Update plane at rate chosen to minimize total work Some projection mandated by Earth curvature Correct for coordinate distortion of each image plane Superior to snapshot imaging and AW Projection Less CPU, memory Diagonal or general Mueller matrices Update or change model in the future as appropriate

15 CPU-based scaling numbers Performance measured by four numbers Can be benchmarked by small programs tconvolve, thogbomclean Numbers shown are from Pawsey Centre ASKAP Real Time Computer 200TF system costing 3.2M Expect SDP to update cost model regularly during pre-construction and construction Track vendors Track scaling work µ wp,µ FFT,µ reproj,µ clean

16 GPU-based scaling numbers Somewhat speculative Grid, clean measured FFT, reproject scaled Substantially better than CPU Use these numbers

17 Data handling Single pass processing is insufficient to meet continuum science goals e.g. dynamic range Algorithms are non-linear Data must be buffered for ~ days to allow multiple calibration passes Require large multi-day visibility data buffer for all telescopes Once continuum sky is well modelled, subtract from spectral visibility

18 Typical computing platform costs Based on GPU scaling Runs all pipelines at full load Mid should have smaller maximum baseline

19 Why is this so good? New algorithm: w snapshots Has better performance and scaling than w projection GPUs have better cost/performance than CPUs used in Pawsey Intel MIC systems may be close Completely untested at the relevant scale SDP consortium will test and extend

20 Caveats Assumes Moore s Law to 2019 Assumes that processing can be scaled up factor of Assumes that we don t need radically new processing True for ASKAP from 2007 to 2013

21 Model for funding compute capacity Initial investment sufficient only for testing, commissioning, and early science Capacity will be incremented as required Part of Board strategy for cost cap

22 Summary Telescope computing capacity will grow as required Scaling work will probably continue into operations Very demanding hands-off processing model Surveys will be possible only after debugging a lot of difficult problems