ASKAP Data Flow ASKAP & MWA Archives Meeting

Size: px
Start display at page:

Download "ASKAP Data Flow ASKAP & MWA Archives Meeting"

Transcription

1 ASKAP Data Flow ASKAP & MWA Archives Meeting Ben Humphreys ASKAP Software and Computing Project Engineer 25 th March 2013 ASTRONOMY AND SPACE SCIENCE

2 Murchison Radioastronomy Observatory

3 Australian SKA Pathfinder (ASKAP) Sited at the Murchison Radio Observatory, Western Australia Observes between 0.7 and 1.8 GHz 36 antennas, 12m diameter Started construction July 2006 Official opening October 2012 Data rate from correlator ~2.5GB/s A DVD every two seconds! Science processing requirement 200TF/s for basic capabilities 600+TF/s for high angular resolution spectral line imaging

4 Phased Array Feeds Key technology development in ASKAP Increases survey speed by approximately an order of magnitude 188 Receiver Elements Typical radio-telescope has two Data-rate ~1.9Tbit/s from each antenna

5 Wide Field of View Image of Centaurus A Required 1200 hours observing on the Australia Telescope Compact Array ASKAP will take about 10 minutes

6 ASKAP Data Flow

7

8 ASKAP Digitiser Card

9

10

11

12 ASKAP DSP Card (Beamformer & Correlator)

13 Custom FPGA Digital Electronics

14 ASKAP Central Processor

15 Key 10 Gigabit Ethernet 4x FDR Infiniband Ingest Pipeline ASKAP Central Processor (aka Pawsey RTC) NOTE: Only primary data network connections are shown. All nodes should be on an appropriate management network too. Data Mover Node 1 Data Mover Node 2 Data Mover Node 3 Data Mover Node 4 Data Mover Node 5 Data Mover Node 6 Cray Cascade Compute Nodes (x484) Sonexion / Lustre Filesystem (1PB) Central Processor Database Nodes (DAE) 8 TB SAS or SSD 8 TB SAS or SSD 8 TB SAS or SSD 8 TB SAS or SSD DB Node 1 DB Node 2 DB Node 3 DB Node 4 4x 10 Gigabit Wavelengths From MRO DWDM Terminal 10GbE Switch Data Mover Node 7 Data Mover Node 8 Data Mover Node 9 FDR Infiniband Fabric (Configured as two switches) Data Mover Node 10 Data Mover Node 11 Data Mover Node 12 Data Mover Node 13 Data Mover Node 14 Cray Cascade Login Nodes Login Node 1 Login Node 2 Data Mover Node 15 Connectivity to MRO Data Mover Node 16 10GbE Switch Traditional Data Mover Role (also warm standby for replacing an ingest node) Data Mover Node 17 Data Mover Node 18 ivec Router/ Firewall 10GBit/s (AARNET) Connectivity to CSIRO corporate network (AARNET) ASKAP Science Operations (Marsfield NSW)

16 Data Conditioner Pipeline (aka Ingest) Ingest Pipeline Telescope Observation Manager Correlator Telescope metadata Visibilities Merge Metadata & Visibilities Flag (From RFI database) Flag (On the fly detection) Apply Calibration Channel Averaging (16200 to 300) Downstream Pipelines Downstream Pipelines Access database of known RFI sources Obtain latest calibration solution Channel Averaging (300 to ~30) Downstream Pipelines Services RFI Source Service Calibration Data Service

17 Processing Pipelines Calibration Pipeline Services Channels (18.5kHz) ccalibrator Calibration Solution Calibration Data Service Large-N (eg. Spectral Line) Imager Pipeline UV Data Channels (18.5kHz) Imager (cimager) Image Cube Source Finder/ Identifier Source Catalog Ingest Pipeline Small-N (e.g. Continuum) Imager Pipeline Sky Model Service UV Data 300 Channels (1MHz) Imager (cimager) Images Source Finder/ Identifier Source Catalog Transient Detector Pipeline ~30 Channels (10MHz) Transient Imager (cfimager) Images Transient Finder/Identifier Transient Detections Light Curve Service

18 Continuum Image

19 Spectral Line Image Cube Image credit: Joint Astronomy Centre, The University of Hawaii

20 Primary Outputs to Science Data Archive Continuum / Polarisation Imaging Pipeline (every 8-12 hours) Images: TB (to HSM) Measurement Set: TB (to HSM) Source Detections: ~30,000 rows (Database Nodes/DAE) Metadata: < 1MB (to Database Nodes/DAE) Spectral Line Imaging Pipeline (every 8-12 hours) Image Cubes: 2.5TB (to HSM) Metadata: < 1MB (to Database Nodes/DAE) Transient Detection Pipeline (every 5 seconds) Flux Measurements: ~1,000 rows (to Database Nodes/DAE) Metadata: < 1MB (to Database Nodes/DAE) Images: variable size/rate (to HSM)

21 Thank you CSIRO Astronomy and Space Science Ben Humphreys ASKAP Computing Project Engineer t e ben.humphreys@csiro.au w ASTRONOMY AND SPACE SCIENCE

ASKAP Central Processor: Design and Implementa8on

ASKAP Central Processor: Design and Implementa8on ASKAP Central Processor: Design and Implementa8on Calibra8on and Imaging Workshop 2014 Ben Humphreys ASKAP So(ware and Compu3ng Project Engineer 3rd - 7th March 2014 ASTRONOMY AND SPACE SCIENCE Australian

More information

CSIRO ASKAP Science Data Archive CSIRO ASTRONOMY AND SPACE SCIENCE (CASS)

CSIRO ASKAP Science Data Archive CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) CSIRO ASKAP Science Data Archive CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) Jessica Chapman, ATUC Meeting, 5 December 2013 CSIRO ASKAP Science Data Archive (CASDA) Talk outline A: CASDA overview and timeline

More information

Signal processing with heterogeneous digital filterbanks: lessons from the MWA and EDA

Signal processing with heterogeneous digital filterbanks: lessons from the MWA and EDA Signal processing with heterogeneous digital filterbanks: lessons from the MWA and EDA Randall Wayth ICRAR/Curtin University with Marcin Sokolowski, Cathryn Trott Outline "Holy grail of CASPER system is

More information

ASKAP Pipeline processing and simulations. Dr Matthew Whiting ASKAP Computing, CSIRO May 5th, 2010

ASKAP Pipeline processing and simulations. Dr Matthew Whiting ASKAP Computing, CSIRO May 5th, 2010 ASKAP Pipeline processing and simulations Dr Matthew Whiting ASKAP Computing, CSIRO May 5th, 2010 ASKAP Computing Team Members Team members Marsfield: Tim Cornwell, Ben Humphreys, Juan Carlos Guzman, Malte

More information

Adaptive selfcalibration for Allen Telescope Array imaging

Adaptive selfcalibration for Allen Telescope Array imaging Adaptive selfcalibration for Allen Telescope Array imaging Garrett Keating, William C. Barott & Melvyn Wright Radio Astronomy laboratory, University of California, Berkeley, CA, 94720 ABSTRACT Planned

More information

SKA Regional Centre Activities in Australasia

SKA Regional Centre Activities in Australasia SKA Regional Centre Activities in Australasia Dr Slava Kitaeff CSIRO-ICRAR APSRC Project Engineer ERIDANUS National Project Lead Why SKA Regional Centres? SKA 1 Observatory Compute capacity: 100 Pflops

More information

Computational issues for HI

Computational issues for HI Computational issues for HI Tim Cornwell, Square Kilometre Array How SKA processes data Science Data Processing system is part of the telescope Only one system per telescope Data flow so large that dedicated

More information

EPICS in the Australian SKA Pathfinder

EPICS in the Australian SKA Pathfinder EPICS in the Australian SKA Pathfinder Craig Haskins Software Engineer 25 April 2012 ASTRONOMY AND SPACE SCIENCE ASKAP Site Murchison Radio Observatory (MRO): Australia s SKA Candidate site Traditional

More information

The Programmable Telescope

The Programmable Telescope The Programmable Telescope Tim Cornwell ASKAP Computing Lead New generation of radio telescopes Flourishing around the world Atacama Large Millimetre Array (ALMA) Expanded Very Large Array (EVLA) Square

More information

VLBI progress Down-under. Tasso Tzioumis Australia Telescope National Facility (ATNF) 25 September 2008

VLBI progress Down-under. Tasso Tzioumis Australia Telescope National Facility (ATNF) 25 September 2008 VLBI progress Down-under Tasso Tzioumis Australia Telescope National Facility (ATNF) 25 September 2008 Outline Down-under == Southern hemisphere VLBI in Australia (LBA) Progress in the last few years Disks

More information

Analysis of the Parallelisation of the Duchamp Algorithm

Analysis of the Parallelisation of the Duchamp Algorithm ivec Research Internships (2009-2010) Analysis of the Parallelisation of the Duchamp Algorithm Stefan Westerlund University of Western Australia Abstract A critical step in radio astronomy is to search

More information

The Square Kilometre Array. Miles Deegan Project Manager, Science Data Processor & Telescope Manager

The Square Kilometre Array. Miles Deegan Project Manager, Science Data Processor & Telescope Manager The Square Kilometre Array Miles Deegan Project Manager, Science Data Processor & Telescope Manager The Square Kilometre Array (SKA) The SKA is a next-generation radio interferometer: 3 telescopes, on

More information

Long Baseline Array Status

Long Baseline Array Status Long Baseline Array Status Cormac Reynolds, Chris Phillips + LBA Team 19 November 2015 CSIRO ASTRONOMY & SPACE SCIENCE LBA LBA VLBI array operated by CSIRO University of Tasmania, Auckland University of

More information

A.J. Faulkner K. Zarb-Adami

A.J. Faulkner K. Zarb-Adami AJ Faulkner K Zarb-Adami March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami SKA1-low requirements (after RBS) Frequency: 50MHz 350MHz Scan angle: >45 Bandwidth: 300MHz # of beams: >5 Sensitivity

More information

PARALLEL PROGRAMMING MANY-CORE COMPUTING: THE LOFAR SOFTWARE TELESCOPE (5/5)

PARALLEL PROGRAMMING MANY-CORE COMPUTING: THE LOFAR SOFTWARE TELESCOPE (5/5) PARALLEL PROGRAMMING MANY-CORE COMPUTING: THE LOFAR SOFTWARE TELESCOPE (5/5) Rob van Nieuwpoort Vrije Universiteit Amsterdam & Astron, the Netherlands Institute for Radio Astronomy Why Radio? Credit: NASA/IPAC

More information

Fast Holographic Deconvolution

Fast Holographic Deconvolution Precision image-domain deconvolution for radio astronomy Ian Sullivan University of Washington 4/19/2013 Precision imaging Modern imaging algorithms grid visibility data using sophisticated beam models

More information

Square Kilometre Array

Square Kilometre Array Square Kilometre Array C4SKA 16 February 2018 David Luchetti Australian SKA Project Director Presentation 1. Video of the Murchison Radioastronomy Observatory 2. Is SKA a Collaborative project? 3. Australia/New

More information

NRAO VLA Archive Survey

NRAO VLA Archive Survey NRAO VLA Archive Survey Jared H. Crossley, Loránt O. Sjouwerman, Edward B. Fomalont, and Nicole M. Radziwill National Radio Astronomy Observatory, 520 Edgemont Road, Charlottesville, Virginia, USA ABSTRACT

More information

CorrelX: A Cloud-Based VLBI Correlator

CorrelX: A Cloud-Based VLBI Correlator CorrelX: A Cloud-Based VLBI Correlator V. Pankratius, A. J. Vazquez, P. Elosegui Massachusetts Institute of Technology Haystack Observatory pankrat@mit.edu, victorpankratius.com 5 th International VLBI

More information

John W. Romein. Netherlands Institute for Radio Astronomy (ASTRON) Dwingeloo, the Netherlands

John W. Romein. Netherlands Institute for Radio Astronomy (ASTRON) Dwingeloo, the Netherlands Signal Processing on GPUs for Radio Telescopes John W. Romein Netherlands Institute for Radio Astronomy (ASTRON) Dwingeloo, the Netherlands 1 Overview radio telescopes six radio telescope algorithms on

More information

Developing A Universal Radio Astronomy Backend. Dr. Ewan Barr, MPIfR Backend Development Group

Developing A Universal Radio Astronomy Backend. Dr. Ewan Barr, MPIfR Backend Development Group Developing A Universal Radio Astronomy Backend Dr. Ewan Barr, MPIfR Backend Development Group Overview Why is it needed? What should it do? Key concepts and technologies Case studies: MeerKAT FBF and APSUSE

More information

SDP Design for Cloudy Regions

SDP Design for Cloudy Regions SDP Design for Cloudy Regions Markus Dolensky, 11/02/2016 2 ICRAR s Data Intensive Astronomy Group M.B. I.C. R.D. M.D. K.V. C.W. A.W. D.P. R.T. generously borrowed content from above colleagues 3 SDP Subelements

More information

AA CORRELATOR SYSTEM CONCEPT DESCRIPTION

AA CORRELATOR SYSTEM CONCEPT DESCRIPTION AA CORRELATOR SYSTEM CONCEPT DESCRIPTION Document number WP2 040.040.010 TD 001 Revision 1 Author. Andrew Faulkner Date.. 2011 03 29 Status.. Approved for release Name Designation Affiliation Date Signature

More information

SKA 1 Infrastructure Element Australia. SKA Engineering Meeting Presentation 8 th October 2013

SKA 1 Infrastructure Element Australia. SKA Engineering Meeting Presentation 8 th October 2013 SKA 1 Infrastructure Element Australia SKA Engineering Meeting Presentation 8 th October 2013 1. Contents Introduction Management Structures Overview Work Breakdown Structure Proposal Review Prototypes

More information

Data Processing for the Square Kilometre Array Telescope

Data Processing for the Square Kilometre Array Telescope Data Processing for the Square Kilometre Array Telescope Streaming Workshop Indianapolis th October Bojan Nikolic Astrophysics Group, Cavendish Lab University of Cambridge Email: b.nikolic@mrao.cam.ac.uk

More information

POSSUM: analysis for early science and beyond

POSSUM: analysis for early science and beyond POSSUM: analysis for early science and beyond Cormac Purcell, Bryan Gaensler and the POSSUM team 2016-06-02 ASKAP 2016 askap.org/possum Rotation 1 Rotation 2 Figure: O'Sullivan et al. 2012 POSSUM-12 versus

More information

The Virtual Observatory in Australia Connecting to International Initiatives. Peter Lamb. CSIRO Mathematical & Information Sciences

The Virtual Observatory in Australia Connecting to International Initiatives. Peter Lamb. CSIRO Mathematical & Information Sciences The Virtual Observatory in Australia Connecting to International Initiatives Peter Lamb CSIRO Mathematical & Information Sciences The Grid & escience Convergence of high-performance computing, huge data

More information

1. ALMA Pipeline Cluster specification. 2. Compute processing node specification: $26K

1. ALMA Pipeline Cluster specification. 2. Compute processing node specification: $26K 1. ALMA Pipeline Cluster specification The following document describes the recommended hardware for the Chilean based cluster for the ALMA pipeline and local post processing to support early science and

More information

Overview of Science Data Processor. Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith

Overview of Science Data Processor. Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith Overview of Science Data Processor Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith Consortium Management and System Engineering Aims of the SDP Work Detailed design of

More information

SKA Computing and Software

SKA Computing and Software SKA Computing and Software Nick Rees 18 May 2016 Summary Introduc)on System overview Compu)ng Elements of the SKA Telescope Manager Low Frequency Aperture Array Central Signal Processor Science Data Processor

More information

SKA Low Correlator & Beamformer - Towards Construction

SKA Low Correlator & Beamformer - Towards Construction SKA Low Correlator & Beamformer - Towards Construction Dr. Grant Hampson 15 th February 2018 ASTRONOMY AND SPACE SCIENCE Presentation Outline Context + Specifications Development team CDR Status + timeline

More information

SKA Technical developments relevant to the National Facility. Keith Grainge University of Manchester

SKA Technical developments relevant to the National Facility. Keith Grainge University of Manchester SKA Technical developments relevant to the National Facility Keith Grainge University of Manchester Talk Overview SKA overview Receptors Data transport and network management Synchronisation and timing

More information

OSKAR-2: Simulating data from the SKA

OSKAR-2: Simulating data from the SKA OSKAR-2: Simulating data from the SKA AACal 2012, Amsterdam, 13 th July 2012 Fred Dulwich, Ben Mort, Stef Salvini 1 Overview OSKAR-2: Interferometer and beamforming simulator package. Intended for simulations

More information

The Canadian CyberSKA Project

The Canadian CyberSKA Project The Canadian CyberSKA Project A. G. Willis (on behalf of the CyberSKA Project Team) National Research Council of Canada Herzberg Institute of Astrophysics Dominion Radio Astrophysical Observatory May 24,

More information

Status hardware, software and what next? Harry Keizer 1, Simon Bijlsma 1, Daniela de Paulis 1,3,Marc Wolf 1,2.

Status hardware, software and what next? Harry Keizer 1, Simon Bijlsma 1, Daniela de Paulis 1,3,Marc Wolf 1,2. SETI@CAMRAS 2017 Status hardware, software and what next? Harry Keizer 1, Simon Bijlsma 1, Daniela de Paulis 1,3,Marc Wolf 1,2 1 Stichting CAMRAS, 2 University of Central Lancashire, 3 University of Plymouth

More information

Imaging and Deconvolution

Imaging and Deconvolution Imaging and Deconvolution Urvashi Rau National Radio Astronomy Observatory, Socorro, NM, USA The van-cittert Zernike theorem Ei E V ij u, v = I l, m e sky j 2 i ul vm dldm 2D Fourier transform : Image

More information

Where is the Slack in the SDP System? Markus Dolensky

Where is the Slack in the SDP System? Markus Dolensky Where is the Slack in the SDP System? Markus Dolensky SKA Signal & Data Path along Design Data Intensive Astronomy LFAA DISH Correlator CSP Science Data Processor 2 SDP Design Challenge revisited Power

More information

SDP Memo 40: PSRFITS Overview for NIP

SDP Memo 40: PSRFITS Overview for NIP SDP Memo 40: PSRFITS Overview for NIP Document Number.......................................................... SDP Memo 40 Document Type.....................................................................

More information

European VLBI Network

European VLBI Network European VLBI Network Cormac Reynolds, JIVE European Radio Interferometry School, Bonn 12 Sept. 2007 EVN Array 15 dissimilar telescopes Observes 3 times a year (approx 60 days per year) Includes some

More information

The Completeness and Reliability of Threshold and False-discovery Rate Source Extraction Algorithms for Compact Continuum Sources

The Completeness and Reliability of Threshold and False-discovery Rate Source Extraction Algorithms for Compact Continuum Sources CSIRO PUBLISHING Publications of the Astronomical Society of Australia, 2012, 29, 229 243 http://dx.doi.org/10.1071/as11026 The Completeness and Reliability of Threshold and False-discovery Rate Source

More information

A ClusterStor update. Torben Kling Petersen, PhD. Principal Architect, HPC

A ClusterStor update. Torben Kling Petersen, PhD. Principal Architect, HPC A ClusterStor update Torben Kling Petersen, PhD Principal Architect, HPC Sonexion (ClusterStor) STILL the fastest file system on the planet!!!! Total system throughput in excess on 1.1 TB/s!! 2 Software

More information

Data Transfer Study for HPSS Archiving

Data Transfer Study for HPSS Archiving Data Transfer Study for HPSS Archiving James R. Wynne III wynnejr@ornl.gov Stanley White whiters@ornl.gov Suzanne T. Parete-Koon paretekoonst@ornl.gov Quinn D. Mitchell mitchellqd@ornl.gov Tom Barron tbarron@ornl.gov

More information

Path to Exascale? Intel in Research and HPC 2012

Path to Exascale? Intel in Research and HPC 2012 Path to Exascale? Intel in Research and HPC 2012 Intel s Investment in Manufacturing New Capacity for 14nm and Beyond D1X Oregon Development Fab Fab 42 Arizona High Volume Fab 22nm Fab Upgrades D1D Oregon

More information

Short Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy

Short Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy Short Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy François Tessier, Venkatram Vishwanath Argonne National Laboratory, USA July 19,

More information

GPU Computing with Fornax. Dr. Christopher Harris

GPU Computing with Fornax. Dr. Christopher Harris GPU Computing with Fornax Dr. Christopher Harris ivec@uwa CAASTRO GPU Training Workshop 8-9 October 2012 Introducing the Historical GPU Graphics Processing Unit (GPU) n : A specialised electronic circuit

More information

PARALLEL PROGRAMMING MANY-CORE COMPUTING FOR THE LOFAR TELESCOPE ROB VAN NIEUWPOORT. Rob van Nieuwpoort

PARALLEL PROGRAMMING MANY-CORE COMPUTING FOR THE LOFAR TELESCOPE ROB VAN NIEUWPOORT. Rob van Nieuwpoort PARALLEL PROGRAMMING MANY-CORE COMPUTING FOR THE LOFAR TELESCOPE ROB VAN NIEUWPOORT Rob van Nieuwpoort rob@cs.vu.nl Who am I 10 years of Grid / Cloud computing 6 years of many-core computing, radio astronomy

More information

MeerKAT Data Architecture. Simon Ratcliffe

MeerKAT Data Architecture. Simon Ratcliffe MeerKAT Data Architecture Simon Ratcliffe MeerKAT Signal Path MeerKAT Data Rates Online System The online system receives raw visibilities from the correlator at a sufficiently high dump rate to facilitate

More information

New Zealand Involvement in Solving the SKA Computing Challenges

New Zealand Involvement in Solving the SKA Computing Challenges New Zealand Involvement in Solving the SKA Computing Challenges D R ANDREW E N S O R D I R ECTO R H P C R ESEARC H L A B O R ATORY/ D I R ECTOR N Z SKA ALLIANCE COMPUTING FO R S K A COLLO Q U I UM 2 0

More information

THE SQUARE KILOMETER ARRAY (SKA) ESD USE CASE

THE SQUARE KILOMETER ARRAY (SKA) ESD USE CASE THE SQUARE KILOMETER ARRAY (SKA) ESD USE CASE Ronald Nijboer Head ASTRON R&D Computing Group With material from Chris Broekema (ASTRON) John Romein (ASTRON) Nick Rees (SKA Office) Miles Deegan (SKA Office)

More information

SKA PHASE 1 CORRELATOR AND TIED ARRAY BEAMFORMER

SKA PHASE 1 CORRELATOR AND TIED ARRAY BEAMFORMER SKA PHASE 1 CORRELATOR AND TIED ARRAY BEAMFORMER Document number... WP2 040.060.010 TD 002 Revision... 1 Author... J. D. Bunton Date... 2011 04 01 Status... Approved for release Name Designation Affiliation

More information

GPU on OpenStack for Science

GPU on OpenStack for Science GPU on OpenStack for Science Deployment and Performance Considerations Luca Cervigni Jeremy Phillips luca.cervigni@pawsey.org.au jeremy.phillips@pawsey.org.au Pawsey Supercomputing Centre Based in Perth,

More information

Powering Real-time Radio Astronomy Signal Processing with latest GPU architectures

Powering Real-time Radio Astronomy Signal Processing with latest GPU architectures Powering Real-time Radio Astronomy Signal Processing with latest GPU architectures Harshavardhan Reddy Suda NCRA, India Vinay Deshpande NVIDIA, India Bharat Kumar NVIDIA, India What signals we are processing?

More information

Memo 126. Strawman SKA Correlator. J.D. Bunton (CSIRO) August

Memo 126. Strawman SKA Correlator. J.D. Bunton (CSIRO) August Memo 126 Strawman SKA Correlator J.D. Bunton (CSIRO) August 2010 www.skatelescope.org/pages/page_memos.htm Enquiries should be addressed to: John.Bunton@csiro.au Document his tory REVISION DATE AUTHOR

More information

Data Handling and Transfer for German LOFAR Stations

Data Handling and Transfer for German LOFAR Stations Data Handling and Transfer for German LOFAR Stations anderson@mpifr-bonn.mpg.de On behalf of LOFAR and GLOW (slides stolen from many other LOFAR presentations by other people) 1/24 Some Initial Comments

More information

The Breakthrough LISTEN Search for Intelligent Life: A Wideband Data Recorder for the Robert C. Byrd Green Bank Telescope

The Breakthrough LISTEN Search for Intelligent Life: A Wideband Data Recorder for the Robert C. Byrd Green Bank Telescope The Breakthrough LISTEN Search for Intelligent Life: A Wideband Data Recorder for the Robert C. Byrd Green Bank Telescope Dave MacMahon University of California at Berkeley Breakthrough LISTEN SETI Project

More information

Compact continuum source finding for next generation radio surveys

Compact continuum source finding for next generation radio surveys Mon. Not. R. Astron. Soc. 422, 1812 1824 (2012) doi:10.1111/j.1365-2966.2012.20768.x Compact continuum source finding for next generation radio surveys P. J. Hancock, 1,2 T. Murphy, 1,2,3 B. M. Gaensler,

More information

ALMA REMOTE MINING EXPERIMENT ARTEMIX. Yaye Awa Ba, Philippe. Salomé, Michel. Caillat (LERMA) with credits to : L. Loria, N.

ALMA REMOTE MINING EXPERIMENT ARTEMIX. Yaye Awa Ba, Philippe. Salomé, Michel. Caillat (LERMA) with credits to : L. Loria, N. ALMA REMOTE MINING EXPERIMENT Yaye Awa Ba, Philippe. Salomé, Michel. Caillat (LERMA) with credits to : L. Loria, N. Kasradze Archive and Data Mining Goals (i) Search by products not by instrumental configuration

More information

Compact continuum source finding for next generation radio surveys

Compact continuum source finding for next generation radio surveys Mon. Not. R. Astron. Soc. 422, 1812 1824 (2012) doi:10.1111/j.1365-2966.2012.20768.x Compact continuum source finding for next generation radio surveys P. J. Hancock, 1,2 T. Murphy, 1,2,3 B. M. Gaensler,

More information

Data Compression for HERA

Data Compression for HERA Data Compression for HERA Paul La Plante, James Aguirre, Bryna Hazelton, Miguel Morales 1. INTRODUCTION The size of the dataset for HERA will soon become intractable for either storage or computing if

More information

EVLA Memo #132 Report on the findings of the CASA Terabyte Initiative: Single-node tests

EVLA Memo #132 Report on the findings of the CASA Terabyte Initiative: Single-node tests EVLA Memo #132 Report on the findings of the CASA Terabyte Initiative: Single-node tests S. Bhatnagar NRAO, Socorro May 18, 2009 Abstract This note reports on the findings of the Terabyte-Initiative of

More information

The Argus 2002 Architecture

The Argus 2002 Architecture 1 The Argus 2002 Architecture Steven W. Ellingson 1 1 The Ohio State University ElectroScience Laboratory, 1320 Kinnear Road, Columbus OH 43212. E-mail: ellingson.1@osu.edu 2 I. Acronyms ANP Argus Narrowband

More information

Data Provenance and Management in Radio Astronomy: A Stream Computing Approach

Data Provenance and Management in Radio Astronomy: A Stream Computing Approach Data Provenance and Management in Radio Astronomy: A Stream Computing Approach Mahmoud S. Mahmoud, Andrew Ensor, Alain Biem, Bruce Elmegreen and Sergei Gulyaev Abstract New approaches for data provenance

More information

THE SKA SCIENCE AND TECHNICAL OPERATIONS PLAN

THE SKA SCIENCE AND TECHNICAL OPERATIONS PLAN THE SKA SCIENCE AND TECHNICAL OPERATIONS PLAN Document number...wp2-001.010.010-pla-001 Revision... A Author... SKA Operations Working Group Date... 2010-01-29 Status... Approved for release Name Designation

More information

LBA evlbi Developments. Chris Phillips evlbi Project Scientist 14 September 2007

LBA evlbi Developments. Chris Phillips evlbi Project Scientist 14 September 2007 LBA evlbi Developments Chris Phillips evlbi Project Scientist 14 September 2007 Summary First science paper Offering via normal call for proposals Partner in EXPReS Fringes to Sh as part of APAN Also Da,

More information

National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology WISE Archive.

National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology WISE Archive. Bruce Berriman / Steve Groom Infrared Science Archive (IRSA), IPAC/Caltech GBB/SLG - 1 WSDC Functional Block Diagram White Sands JPL UCLA HRP H/K MOS Maneuvers SOC Science Team Images FTP Site Ancillary

More information

Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments

Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments Torben Kling-Petersen, PhD Presenter s Name Principle Field Title andengineer Division HPC &Cloud LoB SunComputing Microsystems

More information

PDR.01 SDP Architecture

PDR.01 SDP Architecture PDR.01 SDP Architecture Document number.. SKA-TEL-SDP-0000013 Context.. ARCH Revision...2 Author. Nikolic Release Date 2015-02-09 Document Classification Status. Draft Release Date: 2015-02-09 Page 1 of

More information

The CMS Computing Model

The CMS Computing Model The CMS Computing Model Dorian Kcira California Institute of Technology SuperComputing 2009 November 14-20 2009, Portland, OR CERN s Large Hadron Collider 5000+ Physicists/Engineers 300+ Institutes 70+

More information

The Data exacell DXC. J. Ray Scott DXC PI May 17, 2016

The Data exacell DXC. J. Ray Scott DXC PI May 17, 2016 The Data exacell DXC J. Ray Scott DXC PI May 17, 2016 DXC Leadership Mike Levine Co-Scientific Director Co-PI Nick Nystrom Senior Director of Research Co-PI Ralph Roskies Co-Scientific Director Co-PI Robin

More information

Lessons learnt from implementing mosaicing and faceting in ASKAPsoft. Max Voronkov & Tim Cornwell ASKAP team 2nd April 2009

Lessons learnt from implementing mosaicing and faceting in ASKAPsoft. Max Voronkov & Tim Cornwell ASKAP team 2nd April 2009 Lessons learnt from implementing mosaicing and faceting in ASKAPsoft Max Voronkov & Tim Cornwell ASKAP team 2nd April 2009 Outline - Imaging software ASKAPsoft re-uses LOFAR design Imaging is treated as

More information

HPC NETWORKING IN THE REAL WORLD

HPC NETWORKING IN THE REAL WORLD 15 th ANNUAL WORKSHOP 2019 HPC NETWORKING IN THE REAL WORLD Jesse Martinez Los Alamos National Laboratory March 19 th, 2019 [ LOGO HERE ] LA-UR-19-22146 ABSTRACT Introduction to LANL High Speed Networking

More information

Radio astronomy data reduction at the Institute of Radio Astronomy

Radio astronomy data reduction at the Institute of Radio Astronomy Mem. S.A.It. Suppl. Vol. 13, 79 c SAIt 2009 Memorie della Supplementi Radio astronomy data reduction at the Institute of Radio Astronomy J.S. Morgan INAF Istituto di Radioastronomia, Via P. Gobetti, 101

More information

Data Management. Parallel Filesystems. Dr David Henty HPC Training and Support

Data Management. Parallel Filesystems. Dr David Henty HPC Training and Support Data Management Dr David Henty HPC Training and Support d.henty@epcc.ed.ac.uk +44 131 650 5960 Overview Lecture will cover Why is IO difficult Why is parallel IO even worse Lustre GPFS Performance on ARCHER

More information

Shared Risk Observing

Shared Risk Observing Shared Risk Observing EVLA Advisory Committee Meeting, March 19-20, 2009 Claire Chandler Deputy AD for Science, NM Ops Motivation Early science: Want to get EVLA science capabilities i into the hands of

More information

KVN. Duk-Gyoo Roh, and staffs of KASI & NAOJ + Kagoshima Univ. 8 th VERA User s Meeting, Mitaka NAOJ

KVN. Duk-Gyoo Roh, and staffs of KASI & NAOJ + Kagoshima Univ. 8 th VERA User s Meeting, Mitaka NAOJ KVN Duk-Gyoo Roh, and staffs of KASI & NAOJ + Kagoshima Univ. 8 th VERA User s Meeting, 2010. 09. 16-17. Mitaka NAOJ Contents EAVN & KJJVC project KJJVC framework and status Playback Systems RVDB System

More information

Implementing a Hierarchical Storage Management system in a large-scale Lustre and HPSS environment

Implementing a Hierarchical Storage Management system in a large-scale Lustre and HPSS environment Implementing a Hierarchical Storage Management system in a large-scale Lustre and HPSS environment Brett Bode, Michelle Butler, Sean Stevens, Jim Glasgow National Center for Supercomputing Applications/University

More information

Controlling Field-of-View of Radio Arrays using Weighting Functions

Controlling Field-of-View of Radio Arrays using Weighting Functions Controlling Field-of-View of Radio Arrays using Weighting Functions MIT Haystack FOV Group: Lynn D. Matthews,Colin Lonsdale, Roger Cappallo, Sheperd Doeleman, Divya Oberoi, Vincent Fish Fulfilling scientific

More information

Energy-Efficient Data Transfers in Radio Astronomy with Software UDP RDMA Third Workshop on Innovating the Network for Data-Intensive Science, INDIS16

Energy-Efficient Data Transfers in Radio Astronomy with Software UDP RDMA Third Workshop on Innovating the Network for Data-Intensive Science, INDIS16 Energy-Efficient Data Transfers in Radio Astronomy with Software UDP RDMA Third Workshop on Innovating the Network for Data-Intensive Science, INDIS16 Przemek Lenkiewicz, Researcher@IBM Netherlands Bernard

More information

Correlator Field-of-View Shaping

Correlator Field-of-View Shaping Correlator Field-of-View Shaping Colin Lonsdale Shep Doeleman Vincent Fish Divya Oberoi Lynn Matthews Roger Cappallo Dillon Foight MIT Haystack Observatory Context SKA specifications extremely challenging

More information

MWA Correlator Status. Cambridge MWA Meeting Roger Cappallo

MWA Correlator Status. Cambridge MWA Meeting Roger Cappallo MWA Correlator Status Cambridge MWA Meeting Roger Cappallo 2007.6.25 Correlator Requirements Complex cross-multiply and accumulate data from 524800 signal pairs Each pair comprises 3072 channels with 10

More information

Data Analytics and Storage System (DASS) Mixing POSIX and Hadoop Architectures. 13 November 2016

Data Analytics and Storage System (DASS) Mixing POSIX and Hadoop Architectures. 13 November 2016 National Aeronautics and Space Administration Data Analytics and Storage System (DASS) Mixing POSIX and Hadoop Architectures 13 November 2016 Carrie Spear (carrie.e.spear@nasa.gov) HPC Architect/Contractor

More information

NSF s National Radio Astronomy Observatory (NRAO) and CI. Brian Glendenning Head, NRAO Data Management and Software

NSF s National Radio Astronomy Observatory (NRAO) and CI. Brian Glendenning Head, NRAO Data Management and Software NSF s National Radio Astronomy Observatory (NRAO) and CI Brian Glendenning Head, NRAO Data Management and Software 1 Campus Cyberinfrastructure, Oct 3 2018 Come Visit Us! The Very Large Array (VLA) is

More information

VLA data reduction part 1:

VLA data reduction part 1: VLA data reduction part 1: Post-observing, pre-calibration Loránt Sjouwerman, NRAO Outline After the observations: Obtaining your data from the archive Which CPU processes the data? (Home or NRAO) Examine

More information

TGCC OVERVIEW. 13 février 2014 CEA 10 AVRIL 2012 PAGE 1

TGCC OVERVIEW. 13 février 2014 CEA 10 AVRIL 2012 PAGE 1 STORAGE @ TGCC OVERVIEW CEA 10 AVRIL 2012 PAGE 1 CONTEXT Data-Centric Architecture Centralized storage, accessible from every TGCC s compute machines Make cross-platform data sharing possible Mutualized

More information

Case Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy

Case Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy Case Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy Outline Motivation / Overview Participants / Industry Partners Documentation Architecture Current Status and Services

More information

Data Acquisition. Amedeo Perazzo. SLAC, June 9 th 2009 FAC Review. Photon Controls and Data Systems (PCDS) Group. Amedeo Perazzo

Data Acquisition. Amedeo Perazzo. SLAC, June 9 th 2009 FAC Review. Photon Controls and Data Systems (PCDS) Group. Amedeo Perazzo Data Acquisition Photon Controls and Data Systems (PCDS) Group SLAC, June 9 th 2009 FAC Review 1 Data System Architecture Detector specific Photon Control Data Systems (PCDS) L1: Acquisition Beam Line

More information

Focus Session on Multi-dimensional Data

Focus Session on Multi-dimensional Data Focus Session on Multi-dimensional Data Introduction Mark Allen, Joe Lazio IVOA Interoperability Meeting, ESAC, Madrid, May 20, 2014 CoSADIE Project Science Priority Areas Multi-dimensional Data image:

More information

ARCHER/RDF Overview. How do they fit together? Andy Turner, EPCC

ARCHER/RDF Overview. How do they fit together? Andy Turner, EPCC ARCHER/RDF Overview How do they fit together? Andy Turner, EPCC a.turner@epcc.ed.ac.uk www.epcc.ed.ac.uk www.archer.ac.uk Outline ARCHER/RDF Layout Available file systems Compute resources ARCHER Compute

More information

An ESS implementation in a Tier 1 HPC Centre

An ESS implementation in a Tier 1 HPC Centre An ESS implementation in a Tier 1 HPC Centre Maximising Performance - the NeSI Experience José Higino (NeSI Platforms and NIWA, HPC Systems Engineer) Outline What is NeSI? The National Platforms Framework

More information

Progress towards an Asia-Pacific Regional Centre Network. Peter Quinn (ICRAR)

Progress towards an Asia-Pacific Regional Centre Network. Peter Quinn (ICRAR) Progress towards an Asia-Pacific Regional Centre Network Peter Quinn (ICRAR) What observatories do Regional Centre? Regional Centre Regional Centre? 2 Data rates and volumes ALMA Science Data Products

More information

High dynamic range imaging, computing & I/O load

High dynamic range imaging, computing & I/O load High dynamic range imaging, computing & I/O load RMS ~15µJy/beam RMS ~1µJy/beam S. Bhatnagar NRAO, Socorro Parameterized Measurement Equation Generalized Measurement Equation Obs [ S M V ij = J ij, t W

More information

Data Centres in the Virtual Observatory Age

Data Centres in the Virtual Observatory Age Data Centres in the Virtual Observatory Age David Schade Canadian Astronomy Data Centre A few things I ve learned in the past two days There exist serious efforts at Long-Term Data Preservation Alliance

More information

Argus Radio Telescope Architecture

Argus Radio Telescope Architecture Argus Radio Telescope Architecture Douglas Needham http://cinnion.ka8zrt.com http://www.naapo.org Argus Architecture p.1/15 Introduction: Traditional Telescopes Radio telescopes commonly consist of a single

More information

Self-calibration: about the implementation in GILDAS. Vincent Piétu IRAM. IRAM millimeter interferometry summerschool

Self-calibration: about the implementation in GILDAS. Vincent Piétu IRAM. IRAM millimeter interferometry summerschool Self-calibration: about the implementation in GILDAS Vincent Piétu IRAM 1 About an interferometer sensitivity One usually considers only the noise equation to assess the feasibility of an observation.

More information

DVS, GPFS and External Lustre at NERSC How It s Working on Hopper. Tina Butler, Rei Chi Lee, Gregory Butler 05/25/11 CUG 2011

DVS, GPFS and External Lustre at NERSC How It s Working on Hopper. Tina Butler, Rei Chi Lee, Gregory Butler 05/25/11 CUG 2011 DVS, GPFS and External Lustre at NERSC How It s Working on Hopper Tina Butler, Rei Chi Lee, Gregory Butler 05/25/11 CUG 2011 1 NERSC is the Primary Computing Center for DOE Office of Science NERSC serves

More information

The Murchison Widefield Array Correlator

The Murchison Widefield Array Correlator The Murchison Widefield Array Correlator The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation Ord, S. M., B. Crosse, D. Emrich,

More information

APAN Global Collaboration Linking the World with Light

APAN Global Collaboration Linking the World with Light APAN Global Collaboration Linking the World with Light 1 Andrew Howard Coordinating Engineer Advanced Communications Services and International Development Massive increase in International connectivity

More information

dan.fay@microsoft.com Scientific Data Intensive Computing Workshop 2004 Visualizing and Experiencing E 3 Data + Information: Provide a unique experience to reduce time to insight and knowledge through

More information

Sky-domain algorithms to reconstruct spatial, spectral and time-variable structure of the sky-brightness distribution

Sky-domain algorithms to reconstruct spatial, spectral and time-variable structure of the sky-brightness distribution Sky-domain algorithms to reconstruct spatial, spectral and time-variable structure of the sky-brightness distribution Urvashi Rau National Radio Astronomy Observatory Socorro, NM, USA Outline : - Overview

More information

Concept Design of a Software Correlator for future ALMA. Jongsoo Kim Korea Astronomy and Space Science Institute

Concept Design of a Software Correlator for future ALMA. Jongsoo Kim Korea Astronomy and Space Science Institute Concept Design of a Software Correlator for future ALMA Jongsoo Kim Korea Astronomy and Space Science Institute Technologies for Correlators ASIC (Application-Specific Integrated Circuit) e.g, ALMA 64-antenna

More information