AA CORRELATOR SYSTEM CONCEPT DESCRIPTION

Transcription

1 AA CORRELATOR SYSTEM CONCEPT DESCRIPTION Document number WP TD 001 Revision 1 Author. Andrew Faulkner Date Status.. Approved for release Name Designation Affiliation Date Signature Additional Authors Submitted by: A. Faulkner UCAM Approved by: W. Turner Signal Processing Domain Specialist SPDO

2 DOCUMENT HISTORY Revision Date Of Issue Engineering Change Number Comments A First draft release for internal review 1 29 th March 2011 First Issue DOCUMENT SOFTWARE Package Version Filename Wordprocessor MsWord Word 2003 AA_ Correlator_CoDR_2011 Block diagrams Other ORGANISATION DETAILS Name Physical/Postal Address SKA Program Development Office Jodrell Bank Centre for Astrophysics Alan Turing Building The University of Manchester Oxford Road Manchester, UK M13 9PL Fax. +44 (0) Website Page 2 of 15

3 TABLE OF CONTENTS 1 INTRODUCTION Purpose of the document. 7 2 REFERENCES 7 3 OVERVIEW AA CORRELATOR IMPLEMENTATION Central Processing design Correlator/beamformer C/B Switch for visibility routing and dish corner turning POWER REQUIREMENTS AA CORRELATOR COST Page 3 of 15

4 LIST OF FIGURES Figure 1: SKA Phase 2 overall system diagram from SKADS White Paper. 9 Figure 2: Central processing architectures. 10 Figure 3: A possible physical implementation of AA sub correlator shelf.. 12 LIST OF TABLES Table 1: SKADS SKA implementation 8 Table 2: Power requirement per correlator board. 13 Table 3: Accumulated correlator power.. 14 Table 4: Estimated costs of an AA Correlator for SKA Phase Page 4 of 15

5 LIST OF ABBREVIATIONS AA. Aperture Array ADC. Analogue-to-Digital Converter AI.. Arithmetic Intensity ASIC Application-Specific Integrated Circuit CoDR.. Conceptual Design Review CMAC. Complex Multiplication and ACcumulation CPU. Central Processing Unit CUDA TM. Compute Unified Device Architecture (NVIDIA 2009) DFT. Discrete Fourier Transform DiFX Distributed FX correlator (Deller et al. 2007) DRAM. Dynamic Random Access Memory DRM Design Reference Mission DSP. Digital Signal Processing emerlin.. extended Multi-Element Radio-Linked Interferometer Network evla Extended Very Large Array FFT.. Fast Fourier Transform FLOPS Floating Point Operations per second FPGA.. Field Programmable Gate Array FoV.. Field of View GMRT. Giant Meter-wave Radio Telescope GPGPU. General-Purpose Graphics Processing Unit GPU. Graphics Processing Unit HPC. High-Performance Computing IF.. Intermediate Frequency LO. Local Oscillator LOFAR.. LOw-Frequency ARray MPI.. Message Passing Interface (MPI Forum 2009) MWA Murchison Widefield Array NRE. Non-Recurring Engineering PEP. Project Execution Plan PrepSKA Preparatory Phase for the SKA RF. Radio Frequency SEMP. Systems Engineering Management Plan SRS. Systems Requirement Specification SIMD.. Single Instruction Multiple Data SKA. Square Kilometre Array SKADS.. SKA Design Studies Page 5 of 15

6 SPDO. SKA Program Development Office TBD. To be decided VLBA.. Very Long Baseline Array WIDAR.. Widefield Interferometric Digital ARchitecture (correlator implementation) Page 6 of 15

7 1 Introduction WP TD 001 This paper is a description of a potential correlator structure for an aperture array system. It is not a full design of a correlator; it makes assumptions about the availability processing devices in the timeframe. The design discussed here is structured to support the full AA system for SKA phase 2; it can readily be scaled back for the correlator in phase 1 if necessary. The architecture of an AA correlator is very different from the SKA dish correlator: there are only expected 250 arrays, but each has a very large data output rate of ~16Tb/s. This makes a highly modular and integrated design essential. Many of the issues are related to routing of the large numbers of fibre feeds from each of the arrays. 1.1 Purpose of the document The purpose of this document is to provide a concept description as part of a larger document set in support of the SKA Signal Processing concept design review (CoDR). It provides the concept of how an aperture array correlator could be constructed. System parameters for SKA Phase 2 have been drawn from Schilizzi et al. (2007) [1] and the SKA Design Reference Mission (DRM) [2]. These have been incorporated into the SKADS White Paper, Faulkner et al. [3], much of the design of the correlator has been described in this final report. 2 References [1] Schilizzi, R.T., et al. (2007), Preliminary Specifications for the Square Kilometre Array, Memo 100 [2] SKA Science Working Group, (2010), The Square Kilometre Array Design Reference Mission: SKA mid and SKA lo, report, v1.0, February [3] Faulkner A.J., et al. (2010), Aperture Arrays for the SKA: the SKADS White Paper, Memo 122 [4] Square Kilometre Array Design Studies, SKADS, eu.org [5] Garrett, M.A., et al. (2010), A Concept Design for SKA Phase 1 (SKA 1 ), Memo Page 7 of 15

8 3 Overview WP TD 001 This paper considers the aperture array, AA, correlation requirements for SKA Phase 2. The putative implementation considered is based on the work in the SKA Design Studies, SKADS, [4]. This heavily uses AAs for all frequencies up to 1.4GHz and a substantial dish array from 1.2GHz up to 10 GHz as described in [3]. The outline SKADS SKA implementation is shown in Table 1. The system implementation is shown in Figure 1. Table 1: SKADS SKA implementation Freq. Range Collector Sensitivity Number / size Distribution 70 MHz to 450 MHz 400 MHz to 1.4 GHz Aperture array (AA-low) Aperture array (AA-mid) 4,000 m 2 /K at 100 MHz 10,000 m 2 /K at 800 MHz 250 arrays, Diameter 180 m 250 arrays, Diameter 56 m 66% within core 5 km diameter, rest along 5 spiral arms out to 180 km radius 1.2 GHz to 10 GHz Dishes with wideband single pixel feed (SD-WBSPF) 5,000 m 2 /K at 1.4 GHz 1,200 dishes Diameter 15 m 50% within core 5 km diameter, 25% between the core and 180 km, 25% between 180 km and 3,000 km radius. The requirements of correlation are described elsewhere in this CoDR, the essential difference of the AA correlator becomes clear from examination of an SKA Phase 2 system diagram shown in Figure 1. The correlators receive all the high data rate signal data from all the collectors, cross correlate each baseline of a collector type, integrate and pass the resulting uv data via a large data switch to be buffered at the start of the post processing systems. The data rates from the SKA collectors means that it is impractical to put the correlator anywhere other than local to the core. Assuming that the data rates are reduced sufficiently through the correlator, then it is likely that the uv data will be transported to the nearest major city for processing Page 8 of 15

9 GHz Wide FoV Aperture Array Station Central Processing Facility CPF MHz Wide FoV GHz WB-Single Pixel feeds Dense AA.... Sparse AA Tile & Station Processing DSP Data Time Control To 250 AA Stations 16 Tb/s Optical Data links Pb/s Correlator UV s Image formation Archive AA slice AA slice Dish & AA+Dish Correlation AA slice Data switch Imaging s Tb/s Gb/s Gb/s Control s & User interface Data Archive Science s 15m Dishes DSP 10 Gb/s Time Standard To 1200 Dishes User interface via Internet Figure 1: SKA Phase 2 overall system diagram from SKADS White Paper Page 9 of 15

10 4 AA Correlator implementation WP TD 001 The design of the AA correlator and beamformer cannot at this stage be definitive, since there are too many unknowns before 2018 on, however, the general principles can be discussed and the following is a worked example of a potential implementation. 4.1 Central Processing design Note: The figures here consider the implementation of 2400 dishes. This may be revised for The implementation of the central processor follows from the discussion in the SKADS White Paper [3] discussing imaging and non imaging requirements. The central processing configurations by function are show side by side in Figure 2. UV Image Correlator s formation Beams Visibilities UV data Images Science analysis, user interface & archive Collector Beams Beamforming De-dispersion, retiming, Pulsar spectral separation and Identification profiling Candidates & SKA-Beams Spectra Profiles Science analysis, user interface & archive AA Stations Dishes 250 x 16Tb/s 2400 x 80Gb/s AA slice AA slice Dish & AA+Dish Correlation AA slice Data switch Imaging s Data Archive Science s AA Stations Dishes 250 x 16Tb/s max 2400 x 80Gb/s AA beamformer AA beamformer Dish & AA+Dish Beamformer AA beamformer Data switch Analysis Processing Data Archive Science s Pb/s Tb/s Gb/s Gb/s Pb/s Tb/s Gb/s Gb/s Imaging Figure 2: Central processing architectures Non-imaging The similarity of requirements for the different stages of processing for both systems imply that ensuring that a configuration or alternative programming that can support imaging and non imaging observations will be highly cost effective. Using a unified central processing system avoids extensive data switching for raw collector data; provides opportunities for commensurate observations of imaging and non imaging science experiments; enables innovative new observing techniques to be used; avoids duplicating development effort and is an easier maintenance environment Correlator/beamformer C/B It is assumed that the correlators are FX type and that the frequency division has been done to the appropriate level prior to the beam data arriving at the C/B. This can readily be achieved by the AAs in the station processors. Correlation and beamforming both perform a very large quantity of simple operations, which can be efficiently implemented using integer arithmetic; there is also a great deal of communications I/O. The processing is dominated by complex MACs. Whilst being the same process, there are major structural differences between the correlation of the AA signals and the dish signals. For the AA there are relatively few, ~250, stations each forming very many beams, >1000, over Gb/s data links or equivalent. Whereas, the dishes are providing one beam from up to 2400 collectors over 8 10Gb/s links. This inherently implies that there are two Page 10 of 15

11 C/Bs which confers a number of advantages: all the collectors can be used concurrently; there is no need for large amounts of switching of raw beam data and mass production can be used efficiently for the AA C/B. The number of complex operations, N op cor, for a correlator operating on two polarisations, providing cross polarisation and auto correlation products, for N antenna, is given by: 4 It is convenient to re state this in terms of incoming digital sample rate, data rate G 1 using N bit samples, which covers any mixture of number of beams and bandwidth as with an AA: 1 The AA correlator lends itself to a highly modular implementation split by beams or matched incoming data channels. It is assumed here that the communications are using 10Gb/s (8Gb/s actual data rate with 8:10 encoding) channels, the most cost effective data rate currently and fits conveniently with the available data rates for optical and copper communications. A similar scheme can be built using 100Gb/s channels and splitting the data into channels using a simple switch. Eight of these 10Gb/s channels are colour multiplexed onto a fibre, delivering 80Gb/s per fibre. The AA correlator can then be designed as 200 shelves each with eight identical sub correlators. Each sub correlator can be characterized as having: 250 stations, N 4 bits per sample, N bit Incoming data rate per collector of 8Gb/s, G 1 Hence, the processing rate required per sub correlator is: 63x10 12 complex MACs or ~250TMACs. An outline physical design is shown in Figure 3. This is aimed to make construction and interconnect straightforward. It is constructed as a double sided shelf in a rack, where a multiplexed fibre from each of 250 AA stations is connected using sixteen input cards, each with 16 inputs. It is assumed here that each fibre carries 8 off 10Gb/s channels. The demultiplexed optical signals from the demultiplexer are ordered to match each of the sub processor cards to minimise signal distribution. A 10Gb/s channel from each station is presented to a sub correlator card via a mid plane, which routes the incoming signals appropriately. An alternative connector is the cross connector which links boards orientated at 90 to each other. There are therefore eight sub correlators per shelf. The visibilities are output via local fibre from the sub correlator boards to a data switch for routing to the appropriate UV processor. The maximum correlator output data rate for a continuum survey using 180km baselines and a 100 deg 2 FoV is approximately 14Tb/s [3]. This is a very considerable reduction from the incoming raw data rate of 4Pb/s Page 11 of 15

12 Each sub correlator needs to be capable of processing 250TMACs. In this layout, this could be provided by an array of fifteen of the 20TMAC multi core processing chips, as used in the AAprocessing. The full AA correlator would use 200 of these shelves; at three to a rack the entire system is 70 racks. Improvements in processing performance or communications throughout this sub system could reduce the power or number of boards involved. There are a total of 1600 sub correlator boards in the system. Optical beam inputs 16 cards each: 16inputs of 8x10Gb/s 8 AA slices 8 cards each: 256 inputs of 10Gb/s Beams.. Data Rx & corner turner Visibilities/ Timeseries 256 Fibres.. 1 per station.. 16 per card.. 8 x 10Gb/s ea AA Slices: Correlator/ Beamformer Optical 1:8 demux Optical Rx Midplane Figure 3: A possible physical implementation of AA sub correlator shelf The beamforming requirement for the AAs covering 3 deg 2 is a total of 1.25x10 15 complex MACS. This is a beamforming load of <1T complex MAC per sub correlator; this is much less than the 63T complex MACs required for correlation. The total output data rate from the beamformer of ~3Tb/s for 10,000 beams, 100μS sampling of 8 bits, 2048 channels, and 2 polarisations, each beam is about 330Mb/s. This is rather less than the maximum imaging data rate of 14Tb/s, discussed above, hence the hardware is able to support both beamforming and correlation in the same system Switch for visibility routing and dish corner turning The use of commercial data switch technology is a vital element in this design. The total data rates for the visibility switch and number of ports needed could be required is very large. However, major commercial data centres have similar requirements; hence switches are available with very high performance. In many respects the architecture used here makes the system more straightforward to build: it can be defined at design time what the range of switching that may be required is and the system can then be broken down into a tiered arrangement. The switch is show very simplistically in Figure 2 and in principle can connect any correlator output to any processing blade. This is undoubtedly unnecessary, although the details of the precise flexibility are the subject of substantial development. For example, the correlators could be treated Page 12 of 15

13 completely independently, so that the AA correlator operates only with a defined set of uv processor blades, similarly for the dish correlator. Indeed, the dish correlator splits naturally into a number of frequency bands, which could be independent. These arrangements would be unnecessarily restrictive and careful design should enable data to be piped flexibly between the different subsystems. For this paper it is worth considering the performances available from current equipment. A major determinant is the bisection bandwidth, which is the total data rate that can be moved from one half of the switch to the other. If the bisection bandwidth exceeds the total input data rate then it is likely to be completely non blocking and hence completely flexible. Switches exceeding 50Tb/s 1 are available now, using Infiniband protocols. This compares very favourably with the maximum data rates from the dish correlator of a few hundred Tb/s across the whole band with 2400 dishes and long baselines. The data rate from the AA correlator is significantly lower at ~14Tb/s maximum [3]. The data rate from the ports can be split down to 10Gb/s to match the processing speed of the uv processors. The power requirements are relatively modest with only 8kW being required to fully switch ~50Tb/s. This is currently available; the performance will improve over time with increasing data switching requirements in the commercial sector. 5 Power requirements The total processor cards needed for the correlator is: Aperture Array. 200 sub correlator shelves at eight processing boards per shelf. A total of 1600 correlator cards. The power for the correlator is dominated by the processing chips and their associated communications. In [3], the expectation of a processor of 20TMACs with 128 digital inputs and outputs is discussed in section 7. The nature of the processing device is not specified here, other than it needs to be efficient at the task of correlation and beamforming and be programmable; this is really a discussion on the physical implementation of the system. By using this device the X part of this correlator can have its power requirements estimated. Table 2: Power requirement per correlator board # Function Power ea. (W) Total (W) Remarks 15 Processing devices Processing only. Implemented using the same device as AA DSP Gb/s I/O channel overall I/O channels and on board chip interconnect Electrical power used 475 Electrical power 85% efficiency 560 Cooling power at 25% 140 Board Total 700 Total power used per correlator board 1 The Mellanox IS Page 13 of 15

14 The total correlator power required is shown in Table 3. The estimate, not including switch, is ~1.2MW, which will improve over time with improved devices. Table 3: Accumulated correlator power Type # Power ea. (W) Total (kw) Remarks AA 1600 Processing boards 700 1,120 Board power and interconnect 400,000 Fibre optical receivers , 10Gb/s channels for each 250 AA stations Total AA X correlator power 1,160 6 AA correlator cost The cost of the AA correlator at this stage has to be an estimate with relatively wide errors. However, it is clear that if the design is simplified to this level the minimum cost envelope can be achieved. The programmability of the processing device and its reuse elsewhere in the system avoids the substantial NRE requirements of developing a specialist ASIC. There are, however, significant development costs that are not itemised here; this is the capital cost of implementation. A conservative estimate of the costs is shown in Table 1, which shows a total cost, including fibre receivers, of 45M. This is for the full SKA Phase 2 using 250 AA stations and 16Tb/s per station output data rate. Table 4: Estimated costs of an AA Correlator for SKA Phase 2 Item Quantity Cost each Total Comments Shelf Cost Correlator boards 8 8,000 64,000 Includes cooling, assembly and test Input boards 16 9, ,000 Includes the fibre receivers Backplane 1 2,000 2,000 Mechanics 1 1,000 1,000 Sub-total shelf 211,000 Rack Cost Shelf 3 211, ,000 Rack mechanics 1 1,000 1,000 Power supplies 1 10,000 10,000 Sub-total Rack 643,000 AA-Correlator Rack ,000 45,010, Page 14 of 15

15 The cost of a rack at ~ 600,000 is similar to other high complex processors, giving a simple check. It is interesting to note that the processing itself does not dominate the cost of the correlator, in this implementation with a large number of fibre receivers it is this component which make the biggest cost impact; which may well change with higher speed links Page 15 of 15