LBA evlbi Developments. Chris Phillips evlbi Project Scientist 14 September 2007

Transcription

1 LBA evlbi Developments Chris Phillips evlbi Project Scientist 14 September 2007

2 Summary First science paper Offering via normal call for proposals Partner in EXPReS Fringes to Sh as part of APAN Also Da, Jb, Wb, Tr Looking towards 10+ Gbps using CABB technology

3 ASKAP ATCA Ceduna Mopra Parkes Sydney Tidbinbilla Hobart

4 Network Layout Parkes, ATCA and Mopra each has 2 x 1 Gbps networks connections Links mixed between Sydney and observatories Shared between routed production traffic and layer 2 evlbi data Contract almost signed for 2 x 1 Gbps links between Swinburne and ATNF in Sydney AARNET backbone routed 10 Gbps between capital cities (including Perth) Hobart 1 Gbps connection to university Tasmania to mainland limited by 2x155 Mbps links (shared) Can effectively only use one

5 evlbi Recorder LBADR Based on Metsähovi VSIB DMA card Cousin of EVNPC Highly modified control software Apple Xserve RAID for disk recording Dual onboard Nvidia gigabitethernet Dual port Intel PRO/1000 PCI-Express Benchmarked to 989 Mbps with 20msec RTT 930 Mbps with 55 msec RTT (Sydney-Perth)

6 ATCA Mopra To Seattle 10 Gbps Parkes 10 Gbps AARNET3 To Perth To Swinburne Sydney

7

8 PAMHELA Parkes, ATCA, Mopra, Hobart evlbi Longbaseline array Pa, At, Mp run entirely on ATNF networks Ho over (shared) AARNET routed network Depends on Swinburne DiFX software correlator Currently running on cluster at Parkes 3x256 Mbps (ATCA, Parkes, Mopra) or 4x128 Mbps works flawlessly

9 PAMHELA, v0 Proof of technology Using existing CPSR2 cluster 60 x P4 cores Normally used for pulsar timing Interface via single gateway PC with dual gige network interfaces This machine acts as bottleneck Can utilize 3x 256 Mbps or 4x128 Mbps First science March 2007 Circinus X-1 Phillips, Deller, Amy, Tingay et. al. 2007, MNRAS, 380, L11

10 PAMHELA, v1 Dedicated front end machines to CSPR2 5 dual core, dual processor machines 4 x 1Gbps (local) connections into cluster Parkes limited to 2 x 1Gbps connections Locally implemented using 10 Gbps backbone Will be installed in October Will allow 3x512 Mbps & 128 Mbps from Hobart Plan ATCA-Parkes 1 Gbps before end of 2007

11 PAMHELA, v2-3 Core cluster to be upgraded to 128 cores APSR Install early 2008 Easily handle all data rates, network limited 2 Gbps links to Swinburne allow 3x512 Mbps at Parkes and 3x512 Mbps at Swinburne High sensitivity LBA observations without disk pressure

12 EXPREeS-Oz ATNF and AARNET part of EXPReS project Goal to demonstrate realtime evlbi transfer from Australia to Europe Currently have 1 Gbps lightpath from Sydney-JIVE ATNF - AARNET - CENIC - CANARIE - SurfNet - JIVE Will have 3 lightpaths for 3 x 512 Mbps demonstration in mid October 340 msec RTT

13 EXPREeS-Oz - Implementation On-the-fly translation of LBADR data format to Mark5B Thin Mark5 control emulator TCP-BIC shown to work but slow ramp up speed (>30 sec) causes problems CTCP successfully used Fixed CWND based on RTT and desired bandwidth Developed by Mark Kettenis, JIVE Ramp up within seconds and minimal reduced rate for packet loss UDP transport implemented and works well on formatter tests Simultaneous 512 Mbps on lightpath and 512 Mbps on routed network

14 APAN Participated in APAN evlbi demonstration Mopra connected over lightpath at 256 Mbps ATCA intended to be included also using routed network but bandwidth not available on day Fringes to Sh, Tr, Wb, Jb, Da km baseline Longest evlbi ever (?)

15

16 Network Comparison Three routes to JIVE CSIRO production network via CISCO PIX Direct connection to AARNET3, then routed connection Lightpath from Sydney (layer 2 connection from observatory to Sydney) Very high packet loss at rates > 100 Mbps via firewall. Limits TCP throughput Bypassing firewall much lower packet loss but still limiting TCP to about 400 Mbps < 256 Mbps for APAN demo

17 Lightpath Testing Low packet loss but non-zero Achieve > 800 Mbps sustained TCP-BIC throughout ~30 sec ramp up time Moderately high out-of-order packets Cannot successfully stream multiple connections (or lower bandwidth) over path Approach works well within ATNF networks

18 Network Analysis Tools Iperf Diskspeed Custom built iperf like C program Disk-disk, net-net, net-disk, disk-net, disk-net-disk Getsockopt(TCP_INFO) Returns struct tcp_info Current RTT, cwnd, retransmits, SACKs etc Web100 Web100 metric logging, based on port # Packets out, bytes out, Cwnd, SACKs, DupAcks, RTT, etc

19

20

21

22

23 CABB - Compact Array Broadband Backend New 8 station correlator for ATCA 2 x 2 GHz, dual polarisation 160 Gbps/station In 24U and 6.6 kw max power dissipation For VLBI usable as 2 GHz dual polarisation Need up to 80 Gbps per station Realistically Gbps All channelization at station Thousands of channels for spectral line experiments Fringe rotate at station, then convert to 2 bit complex

24 ATNF Chris Phillips evlbi Project Scientist Phone: Chris.Phillips@csiro.au Web: Thank you Contact Us Phone: or enquiries@csiro.au Web: