A Scalable, FPGA Based 8 Station Correlator Based on Modular Hardware and Parameterized Libraries

Transcription

1 A Scalable, FPGA Based 8 Station Correlator Based on Modular Hardware and Parameterized Libraries Aaron Parsons Space Sciences Lab University of California, Berkeley

2 CASPER: Center for Astronomy Signal Processing and Electronics Research Don Backer, Chen Chang, Henry Chen, Daniel Chapman, Pat Crescini, Christina DeJesus, Pierre Droz, David McMahon, Jeff Mock, Aaron Parsons, Andrew Siemion, Dan Werthimer, Mel Wright Radio Astronomy Lab Leo Blitz, Dave Deboer, Paul Demorest, Matt Dexter, John Dreher, Carl Heiles, Jack Welch, Lynn Urry Berkeley Wireless Research Center Bob Broderson, John Wawrzynek NRAO/UVA Charlottesville Rich Bradley, Nicole Gugliucci, Chiatali Parashare

3 The Problem with the Current Hardware Development Model Takes 5 years Cost Dominated by NRE because of custom: Boards, Backplanes, Protocols Antiquated by the time it s released.

4 Solution for Real-time Signal Processing: Modular, upgradeable hardware Industry standard communication protocols Reusable, platform-independent gateware

5 Modular Hardware: Low number of board designs Can be upgraded piecemeal or all together Reusable Standard signal processing model which is consistent between upgrades.

6

7 SETI Applications JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth) Parkes Southern SERENDIP ALFA Sky Survey (300 MHz x 7 beams) SETI Italia (Bologna) SETI@home Astronomy Applications GALFA Spectrometer Arecibo Multibeam Hydrogen Survey Astronomy Signal Processor ASP Don Backer, Ingrid Stairs, (pulsars) ATA4 Correlator F Engine Reionization Experiments (Backer (UCB), Chippendale/Ekers Antenna Holography, ATNF, China (ATNF))

8

9

10

11 BEE2 Compute Module Diagram 4GB DDR2 DRAM 12.8GB/s (400DDR) Memory Controller FPGA Fabric FPGA Fabric b 64 MGT 5 FPGAs 2VP70FF1704 IB4X/CX4 40Gbps MGT MGT DRAM DRAM IB4X/CX4 20Gbps 100BT Ethernet DRAM Memory Controller DRAM DRAM DRAM DRAM DRAM IB4X/CX4 40Gbps R DD 0 30 FPGA Fabric Memory Controller DRAM DRAM DRAM DRAM IB4X/CX4 40Gbps IB4X/CX4 40Gbps FPGA Fabric MGT MGT FPGA Fabric 138 bits 300MHz DDR 41.4Gb/s Memory Controller Memory Controller DRAM DRAM DRAM DRAM DRAM DRAM DRAM DRAM

12 Global Interconnects Ethernet Switch Commercial Infiniband switch from Mellanox, Voltaire, etc. Packet switched, non-blocking 24 ~ 144 ports (4X) per chassis Compute Node #1 Compute Node #N Up to 10,000 ports in a system 200~1000 ns switch latency 400~1200 ns FPGA to FPGA latency 480Gbps ~ 2.88Tbps full duplex constant cross section bandwidth <$400 per port Infiniband Crossbar Switch

13 Platform-Independent, Parameterized Gateware What is Gateware? Design logic of FPGAs (between hardware and software) Need libraries for signal processing which don t have to be rewritten every hardware generation. Simulink!

14 Matlab/Simulink/System Generator

15 Biplex Pipelined FFT Uses 1/6 the resources of the Xilinx module. Performs 2 complex FFT s for the price of one.

16 Wide-band FFT: Beyond the system clock biplex... direct

17 FFT Controls

18

19 Polyphase Filter Banks

20 Filter Response: PFB vs. FFT

21 PFB vs. FFT

22 Additional PFB controls Filter overlap Width of filter coefficients Window function for filter (hamming, hanning, etc.) Import filter coefficients for custom filter performance Both FFT and PFB available as Verilog modules (no proprietary software, but not as portable between chips/architectures).

23 Digital Down-Converter Selectable # of FIR taps On-the-fly programmable mix frequency Selectable FIR coeff Agile sub-band selection.

24 X-Engine Correlation Architecture

25 X-Engine Architecture: applied to an arbitrary sized antenna array

26 Applications VLBI Mark 5B data recorder - Haystack 500 MHz VLBA and Beamforming - CfA, Bob Wilson, Jonathan Weintroub SETI Arecibo (UCB) ATA (Seti Institute) JPL/UCB/SI DSN 20 GHz dual pol (Preston, Gulkis, Levin, Jones) Correlators and Imagers: ATA (Mel Wright) Reionization Experiment (Backer/NRAO) Carma Next Gen (Dave Hawkins, Caltech) SKA demonstrator South Africa (Justin Jonas) MWAR, LWA MIT, NRL

27 Gluing the System Together Debian linux running on the BEE2 FPGAs Interfaces with FPGA fabric appearing in /dev under linux TCP/IP server running on IBOB NFS mounting of off-board storage Hardware device abstraction under Simulink. Unpluggable, self-synchronizing interconnect

28 CASPER the Friendly... Group Helping Open-source Signalprocessing Technology (GHOST?) Goal to help develop signal processing instrumenation and libraries for the community. Open source hardware, gateware, and software. Provide training and tutorials Not so much delivering turn-key instruments. Supported by NSF ATI Grant

29 LEGO Blocks = BEE2 = ADC = 10 GbE cable = IBOB = Data Selection = X Engine (correlator) = Filter = Mixer = Switch = Transposer = PFB/FFT = Buffer = Integrator

30 Portable VLBI backend Interfaces to MARK 5B data recorder 500 MHz spectrum recorder. (This makes 4 instruments in 1 year!)

31 Pocket Correlator 2 dual polarization antennas 512 channels

32 Phase I: FX, 8 antenna, dual polarization, 150 MHz bandwidth, 256 channels, 4 bit correlation Done!

33 Phase II: FX, 16 antenna, dual polarization, 150 MHz bandwidth, 2048 channels, 4 bit correlation Features: - Transpose in SRAM - Accumulation in DRAM - Wideband (500 MHz) option. 50% Done

34 Phase III: Generic Correlator FX, Flexible Bandwidth, Tons of Antennas, Computing-by-the-yard

35 Beowulf-like Computing Architecture Doesn t have to be FPGA-based

36 The Current PAPER Correlator: 8 antenna, dual polarization, 150 MHz Bandwidth

37

38 8-Station, IBOB design

39 8-Station, Corner FPGA Design

40 8-Station, Center FPGA Design

41

42

43

44 Movie of First Light: 8-station Snapshot images (from 8 second integrations) Instantaneous baselines (no earth rotation) Multi-frequency image synthesis Raw image (not deconvolved by beam pattern) Caveat: Not indicative of actual imaging power of correlator...