ASKAP Data Flow ASKAP & MWA Archives Meeting
|
|
- Anastasia Kennedy
- 5 years ago
- Views:
Transcription
1 ASKAP Data Flow ASKAP & MWA Archives Meeting Ben Humphreys ASKAP Software and Computing Project Engineer 25 th March 2013 ASTRONOMY AND SPACE SCIENCE
2 Murchison Radioastronomy Observatory
3 Australian SKA Pathfinder (ASKAP) Sited at the Murchison Radio Observatory, Western Australia Observes between 0.7 and 1.8 GHz 36 antennas, 12m diameter Started construction July 2006 Official opening October 2012 Data rate from correlator ~2.5GB/s A DVD every two seconds! Science processing requirement 200TF/s for basic capabilities 600+TF/s for high angular resolution spectral line imaging
4 Phased Array Feeds Key technology development in ASKAP Increases survey speed by approximately an order of magnitude 188 Receiver Elements Typical radio-telescope has two Data-rate ~1.9Tbit/s from each antenna
5 Wide Field of View Image of Centaurus A Required 1200 hours observing on the Australia Telescope Compact Array ASKAP will take about 10 minutes
6 ASKAP Data Flow
7
8 ASKAP Digitiser Card
9
10
11
12 ASKAP DSP Card (Beamformer & Correlator)
13 Custom FPGA Digital Electronics
14 ASKAP Central Processor
15 Key 10 Gigabit Ethernet 4x FDR Infiniband Ingest Pipeline ASKAP Central Processor (aka Pawsey RTC) NOTE: Only primary data network connections are shown. All nodes should be on an appropriate management network too. Data Mover Node 1 Data Mover Node 2 Data Mover Node 3 Data Mover Node 4 Data Mover Node 5 Data Mover Node 6 Cray Cascade Compute Nodes (x484) Sonexion / Lustre Filesystem (1PB) Central Processor Database Nodes (DAE) 8 TB SAS or SSD 8 TB SAS or SSD 8 TB SAS or SSD 8 TB SAS or SSD DB Node 1 DB Node 2 DB Node 3 DB Node 4 4x 10 Gigabit Wavelengths From MRO DWDM Terminal 10GbE Switch Data Mover Node 7 Data Mover Node 8 Data Mover Node 9 FDR Infiniband Fabric (Configured as two switches) Data Mover Node 10 Data Mover Node 11 Data Mover Node 12 Data Mover Node 13 Data Mover Node 14 Cray Cascade Login Nodes Login Node 1 Login Node 2 Data Mover Node 15 Connectivity to MRO Data Mover Node 16 10GbE Switch Traditional Data Mover Role (also warm standby for replacing an ingest node) Data Mover Node 17 Data Mover Node 18 ivec Router/ Firewall 10GBit/s (AARNET) Connectivity to CSIRO corporate network (AARNET) ASKAP Science Operations (Marsfield NSW)
16 Data Conditioner Pipeline (aka Ingest) Ingest Pipeline Telescope Observation Manager Correlator Telescope metadata Visibilities Merge Metadata & Visibilities Flag (From RFI database) Flag (On the fly detection) Apply Calibration Channel Averaging (16200 to 300) Downstream Pipelines Downstream Pipelines Access database of known RFI sources Obtain latest calibration solution Channel Averaging (300 to ~30) Downstream Pipelines Services RFI Source Service Calibration Data Service
17 Processing Pipelines Calibration Pipeline Services Channels (18.5kHz) ccalibrator Calibration Solution Calibration Data Service Large-N (eg. Spectral Line) Imager Pipeline UV Data Channels (18.5kHz) Imager (cimager) Image Cube Source Finder/ Identifier Source Catalog Ingest Pipeline Small-N (e.g. Continuum) Imager Pipeline Sky Model Service UV Data 300 Channels (1MHz) Imager (cimager) Images Source Finder/ Identifier Source Catalog Transient Detector Pipeline ~30 Channels (10MHz) Transient Imager (cfimager) Images Transient Finder/Identifier Transient Detections Light Curve Service
18 Continuum Image
19 Spectral Line Image Cube Image credit: Joint Astronomy Centre, The University of Hawaii
20 Primary Outputs to Science Data Archive Continuum / Polarisation Imaging Pipeline (every 8-12 hours) Images: TB (to HSM) Measurement Set: TB (to HSM) Source Detections: ~30,000 rows (Database Nodes/DAE) Metadata: < 1MB (to Database Nodes/DAE) Spectral Line Imaging Pipeline (every 8-12 hours) Image Cubes: 2.5TB (to HSM) Metadata: < 1MB (to Database Nodes/DAE) Transient Detection Pipeline (every 5 seconds) Flux Measurements: ~1,000 rows (to Database Nodes/DAE) Metadata: < 1MB (to Database Nodes/DAE) Images: variable size/rate (to HSM)
21 Thank you CSIRO Astronomy and Space Science Ben Humphreys ASKAP Computing Project Engineer t e ben.humphreys@csiro.au w ASTRONOMY AND SPACE SCIENCE
ASKAP Central Processor: Design and Implementa8on
ASKAP Central Processor: Design and Implementa8on Calibra8on and Imaging Workshop 2014 Ben Humphreys ASKAP So(ware and Compu3ng Project Engineer 3rd - 7th March 2014 ASTRONOMY AND SPACE SCIENCE Australian
More informationCSIRO ASKAP Science Data Archive CSIRO ASTRONOMY AND SPACE SCIENCE (CASS)
CSIRO ASKAP Science Data Archive CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) Jessica Chapman, ATUC Meeting, 5 December 2013 CSIRO ASKAP Science Data Archive (CASDA) Talk outline A: CASDA overview and timeline
More informationSignal processing with heterogeneous digital filterbanks: lessons from the MWA and EDA
Signal processing with heterogeneous digital filterbanks: lessons from the MWA and EDA Randall Wayth ICRAR/Curtin University with Marcin Sokolowski, Cathryn Trott Outline "Holy grail of CASPER system is
More informationASKAP Pipeline processing and simulations. Dr Matthew Whiting ASKAP Computing, CSIRO May 5th, 2010
ASKAP Pipeline processing and simulations Dr Matthew Whiting ASKAP Computing, CSIRO May 5th, 2010 ASKAP Computing Team Members Team members Marsfield: Tim Cornwell, Ben Humphreys, Juan Carlos Guzman, Malte
More informationAdaptive selfcalibration for Allen Telescope Array imaging
Adaptive selfcalibration for Allen Telescope Array imaging Garrett Keating, William C. Barott & Melvyn Wright Radio Astronomy laboratory, University of California, Berkeley, CA, 94720 ABSTRACT Planned
More informationSKA Regional Centre Activities in Australasia
SKA Regional Centre Activities in Australasia Dr Slava Kitaeff CSIRO-ICRAR APSRC Project Engineer ERIDANUS National Project Lead Why SKA Regional Centres? SKA 1 Observatory Compute capacity: 100 Pflops
More informationComputational issues for HI
Computational issues for HI Tim Cornwell, Square Kilometre Array How SKA processes data Science Data Processing system is part of the telescope Only one system per telescope Data flow so large that dedicated
More informationEPICS in the Australian SKA Pathfinder
EPICS in the Australian SKA Pathfinder Craig Haskins Software Engineer 25 April 2012 ASTRONOMY AND SPACE SCIENCE ASKAP Site Murchison Radio Observatory (MRO): Australia s SKA Candidate site Traditional
More informationThe Programmable Telescope
The Programmable Telescope Tim Cornwell ASKAP Computing Lead New generation of radio telescopes Flourishing around the world Atacama Large Millimetre Array (ALMA) Expanded Very Large Array (EVLA) Square
More informationVLBI progress Down-under. Tasso Tzioumis Australia Telescope National Facility (ATNF) 25 September 2008
VLBI progress Down-under Tasso Tzioumis Australia Telescope National Facility (ATNF) 25 September 2008 Outline Down-under == Southern hemisphere VLBI in Australia (LBA) Progress in the last few years Disks
More informationAnalysis of the Parallelisation of the Duchamp Algorithm
ivec Research Internships (2009-2010) Analysis of the Parallelisation of the Duchamp Algorithm Stefan Westerlund University of Western Australia Abstract A critical step in radio astronomy is to search
More informationThe Square Kilometre Array. Miles Deegan Project Manager, Science Data Processor & Telescope Manager
The Square Kilometre Array Miles Deegan Project Manager, Science Data Processor & Telescope Manager The Square Kilometre Array (SKA) The SKA is a next-generation radio interferometer: 3 telescopes, on
More informationLong Baseline Array Status
Long Baseline Array Status Cormac Reynolds, Chris Phillips + LBA Team 19 November 2015 CSIRO ASTRONOMY & SPACE SCIENCE LBA LBA VLBI array operated by CSIRO University of Tasmania, Auckland University of
More informationA.J. Faulkner K. Zarb-Adami
AJ Faulkner K Zarb-Adami March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami SKA1-low requirements (after RBS) Frequency: 50MHz 350MHz Scan angle: >45 Bandwidth: 300MHz # of beams: >5 Sensitivity
More informationPARALLEL PROGRAMMING MANY-CORE COMPUTING: THE LOFAR SOFTWARE TELESCOPE (5/5)
PARALLEL PROGRAMMING MANY-CORE COMPUTING: THE LOFAR SOFTWARE TELESCOPE (5/5) Rob van Nieuwpoort Vrije Universiteit Amsterdam & Astron, the Netherlands Institute for Radio Astronomy Why Radio? Credit: NASA/IPAC
More informationFast Holographic Deconvolution
Precision image-domain deconvolution for radio astronomy Ian Sullivan University of Washington 4/19/2013 Precision imaging Modern imaging algorithms grid visibility data using sophisticated beam models
More informationSquare Kilometre Array
Square Kilometre Array C4SKA 16 February 2018 David Luchetti Australian SKA Project Director Presentation 1. Video of the Murchison Radioastronomy Observatory 2. Is SKA a Collaborative project? 3. Australia/New
More informationNRAO VLA Archive Survey
NRAO VLA Archive Survey Jared H. Crossley, Loránt O. Sjouwerman, Edward B. Fomalont, and Nicole M. Radziwill National Radio Astronomy Observatory, 520 Edgemont Road, Charlottesville, Virginia, USA ABSTRACT
More informationCorrelX: A Cloud-Based VLBI Correlator
CorrelX: A Cloud-Based VLBI Correlator V. Pankratius, A. J. Vazquez, P. Elosegui Massachusetts Institute of Technology Haystack Observatory pankrat@mit.edu, victorpankratius.com 5 th International VLBI
More informationJohn W. Romein. Netherlands Institute for Radio Astronomy (ASTRON) Dwingeloo, the Netherlands
Signal Processing on GPUs for Radio Telescopes John W. Romein Netherlands Institute for Radio Astronomy (ASTRON) Dwingeloo, the Netherlands 1 Overview radio telescopes six radio telescope algorithms on
More informationDeveloping A Universal Radio Astronomy Backend. Dr. Ewan Barr, MPIfR Backend Development Group
Developing A Universal Radio Astronomy Backend Dr. Ewan Barr, MPIfR Backend Development Group Overview Why is it needed? What should it do? Key concepts and technologies Case studies: MeerKAT FBF and APSUSE
More informationSDP Design for Cloudy Regions
SDP Design for Cloudy Regions Markus Dolensky, 11/02/2016 2 ICRAR s Data Intensive Astronomy Group M.B. I.C. R.D. M.D. K.V. C.W. A.W. D.P. R.T. generously borrowed content from above colleagues 3 SDP Subelements
More informationAA CORRELATOR SYSTEM CONCEPT DESCRIPTION
AA CORRELATOR SYSTEM CONCEPT DESCRIPTION Document number WP2 040.040.010 TD 001 Revision 1 Author. Andrew Faulkner Date.. 2011 03 29 Status.. Approved for release Name Designation Affiliation Date Signature
More informationSKA 1 Infrastructure Element Australia. SKA Engineering Meeting Presentation 8 th October 2013
SKA 1 Infrastructure Element Australia SKA Engineering Meeting Presentation 8 th October 2013 1. Contents Introduction Management Structures Overview Work Breakdown Structure Proposal Review Prototypes
More informationData Processing for the Square Kilometre Array Telescope
Data Processing for the Square Kilometre Array Telescope Streaming Workshop Indianapolis th October Bojan Nikolic Astrophysics Group, Cavendish Lab University of Cambridge Email: b.nikolic@mrao.cam.ac.uk
More informationPOSSUM: analysis for early science and beyond
POSSUM: analysis for early science and beyond Cormac Purcell, Bryan Gaensler and the POSSUM team 2016-06-02 ASKAP 2016 askap.org/possum Rotation 1 Rotation 2 Figure: O'Sullivan et al. 2012 POSSUM-12 versus
More informationThe Virtual Observatory in Australia Connecting to International Initiatives. Peter Lamb. CSIRO Mathematical & Information Sciences
The Virtual Observatory in Australia Connecting to International Initiatives Peter Lamb CSIRO Mathematical & Information Sciences The Grid & escience Convergence of high-performance computing, huge data
More information1. ALMA Pipeline Cluster specification. 2. Compute processing node specification: $26K
1. ALMA Pipeline Cluster specification The following document describes the recommended hardware for the Chilean based cluster for the ALMA pipeline and local post processing to support early science and
More informationOverview of Science Data Processor. Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith
Overview of Science Data Processor Paul Alexander Rosie Bolton Ian Cooper Bojan Nikolic Simon Ratcliffe Harry Smith Consortium Management and System Engineering Aims of the SDP Work Detailed design of
More informationSKA Computing and Software
SKA Computing and Software Nick Rees 18 May 2016 Summary Introduc)on System overview Compu)ng Elements of the SKA Telescope Manager Low Frequency Aperture Array Central Signal Processor Science Data Processor
More informationSKA Low Correlator & Beamformer - Towards Construction
SKA Low Correlator & Beamformer - Towards Construction Dr. Grant Hampson 15 th February 2018 ASTRONOMY AND SPACE SCIENCE Presentation Outline Context + Specifications Development team CDR Status + timeline
More informationSKA Technical developments relevant to the National Facility. Keith Grainge University of Manchester
SKA Technical developments relevant to the National Facility Keith Grainge University of Manchester Talk Overview SKA overview Receptors Data transport and network management Synchronisation and timing
More informationOSKAR-2: Simulating data from the SKA
OSKAR-2: Simulating data from the SKA AACal 2012, Amsterdam, 13 th July 2012 Fred Dulwich, Ben Mort, Stef Salvini 1 Overview OSKAR-2: Interferometer and beamforming simulator package. Intended for simulations
More informationThe Canadian CyberSKA Project
The Canadian CyberSKA Project A. G. Willis (on behalf of the CyberSKA Project Team) National Research Council of Canada Herzberg Institute of Astrophysics Dominion Radio Astrophysical Observatory May 24,
More informationStatus hardware, software and what next? Harry Keizer 1, Simon Bijlsma 1, Daniela de Paulis 1,3,Marc Wolf 1,2.
SETI@CAMRAS 2017 Status hardware, software and what next? Harry Keizer 1, Simon Bijlsma 1, Daniela de Paulis 1,3,Marc Wolf 1,2 1 Stichting CAMRAS, 2 University of Central Lancashire, 3 University of Plymouth
More informationImaging and Deconvolution
Imaging and Deconvolution Urvashi Rau National Radio Astronomy Observatory, Socorro, NM, USA The van-cittert Zernike theorem Ei E V ij u, v = I l, m e sky j 2 i ul vm dldm 2D Fourier transform : Image
More informationWhere is the Slack in the SDP System? Markus Dolensky
Where is the Slack in the SDP System? Markus Dolensky SKA Signal & Data Path along Design Data Intensive Astronomy LFAA DISH Correlator CSP Science Data Processor 2 SDP Design Challenge revisited Power
More informationSDP Memo 40: PSRFITS Overview for NIP
SDP Memo 40: PSRFITS Overview for NIP Document Number.......................................................... SDP Memo 40 Document Type.....................................................................
More informationEuropean VLBI Network
European VLBI Network Cormac Reynolds, JIVE European Radio Interferometry School, Bonn 12 Sept. 2007 EVN Array 15 dissimilar telescopes Observes 3 times a year (approx 60 days per year) Includes some
More informationThe Completeness and Reliability of Threshold and False-discovery Rate Source Extraction Algorithms for Compact Continuum Sources
CSIRO PUBLISHING Publications of the Astronomical Society of Australia, 2012, 29, 229 243 http://dx.doi.org/10.1071/as11026 The Completeness and Reliability of Threshold and False-discovery Rate Source
More informationA ClusterStor update. Torben Kling Petersen, PhD. Principal Architect, HPC
A ClusterStor update Torben Kling Petersen, PhD Principal Architect, HPC Sonexion (ClusterStor) STILL the fastest file system on the planet!!!! Total system throughput in excess on 1.1 TB/s!! 2 Software
More informationData Transfer Study for HPSS Archiving
Data Transfer Study for HPSS Archiving James R. Wynne III wynnejr@ornl.gov Stanley White whiters@ornl.gov Suzanne T. Parete-Koon paretekoonst@ornl.gov Quinn D. Mitchell mitchellqd@ornl.gov Tom Barron tbarron@ornl.gov
More informationPath to Exascale? Intel in Research and HPC 2012
Path to Exascale? Intel in Research and HPC 2012 Intel s Investment in Manufacturing New Capacity for 14nm and Beyond D1X Oregon Development Fab Fab 42 Arizona High Volume Fab 22nm Fab Upgrades D1D Oregon
More informationShort Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy
Short Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy François Tessier, Venkatram Vishwanath Argonne National Laboratory, USA July 19,
More informationGPU Computing with Fornax. Dr. Christopher Harris
GPU Computing with Fornax Dr. Christopher Harris ivec@uwa CAASTRO GPU Training Workshop 8-9 October 2012 Introducing the Historical GPU Graphics Processing Unit (GPU) n : A specialised electronic circuit
More informationPARALLEL PROGRAMMING MANY-CORE COMPUTING FOR THE LOFAR TELESCOPE ROB VAN NIEUWPOORT. Rob van Nieuwpoort
PARALLEL PROGRAMMING MANY-CORE COMPUTING FOR THE LOFAR TELESCOPE ROB VAN NIEUWPOORT Rob van Nieuwpoort rob@cs.vu.nl Who am I 10 years of Grid / Cloud computing 6 years of many-core computing, radio astronomy
More informationMeerKAT Data Architecture. Simon Ratcliffe
MeerKAT Data Architecture Simon Ratcliffe MeerKAT Signal Path MeerKAT Data Rates Online System The online system receives raw visibilities from the correlator at a sufficiently high dump rate to facilitate
More informationNew Zealand Involvement in Solving the SKA Computing Challenges
New Zealand Involvement in Solving the SKA Computing Challenges D R ANDREW E N S O R D I R ECTO R H P C R ESEARC H L A B O R ATORY/ D I R ECTOR N Z SKA ALLIANCE COMPUTING FO R S K A COLLO Q U I UM 2 0
More informationTHE SQUARE KILOMETER ARRAY (SKA) ESD USE CASE
THE SQUARE KILOMETER ARRAY (SKA) ESD USE CASE Ronald Nijboer Head ASTRON R&D Computing Group With material from Chris Broekema (ASTRON) John Romein (ASTRON) Nick Rees (SKA Office) Miles Deegan (SKA Office)
More informationSKA PHASE 1 CORRELATOR AND TIED ARRAY BEAMFORMER
SKA PHASE 1 CORRELATOR AND TIED ARRAY BEAMFORMER Document number... WP2 040.060.010 TD 002 Revision... 1 Author... J. D. Bunton Date... 2011 04 01 Status... Approved for release Name Designation Affiliation
More informationGPU on OpenStack for Science
GPU on OpenStack for Science Deployment and Performance Considerations Luca Cervigni Jeremy Phillips luca.cervigni@pawsey.org.au jeremy.phillips@pawsey.org.au Pawsey Supercomputing Centre Based in Perth,
More informationPowering Real-time Radio Astronomy Signal Processing with latest GPU architectures
Powering Real-time Radio Astronomy Signal Processing with latest GPU architectures Harshavardhan Reddy Suda NCRA, India Vinay Deshpande NVIDIA, India Bharat Kumar NVIDIA, India What signals we are processing?
More informationMemo 126. Strawman SKA Correlator. J.D. Bunton (CSIRO) August
Memo 126 Strawman SKA Correlator J.D. Bunton (CSIRO) August 2010 www.skatelescope.org/pages/page_memos.htm Enquiries should be addressed to: John.Bunton@csiro.au Document his tory REVISION DATE AUTHOR
More informationData Handling and Transfer for German LOFAR Stations
Data Handling and Transfer for German LOFAR Stations anderson@mpifr-bonn.mpg.de On behalf of LOFAR and GLOW (slides stolen from many other LOFAR presentations by other people) 1/24 Some Initial Comments
More informationThe Breakthrough LISTEN Search for Intelligent Life: A Wideband Data Recorder for the Robert C. Byrd Green Bank Telescope
The Breakthrough LISTEN Search for Intelligent Life: A Wideband Data Recorder for the Robert C. Byrd Green Bank Telescope Dave MacMahon University of California at Berkeley Breakthrough LISTEN SETI Project
More informationCompact continuum source finding for next generation radio surveys
Mon. Not. R. Astron. Soc. 422, 1812 1824 (2012) doi:10.1111/j.1365-2966.2012.20768.x Compact continuum source finding for next generation radio surveys P. J. Hancock, 1,2 T. Murphy, 1,2,3 B. M. Gaensler,
More informationALMA REMOTE MINING EXPERIMENT ARTEMIX. Yaye Awa Ba, Philippe. Salomé, Michel. Caillat (LERMA) with credits to : L. Loria, N.
ALMA REMOTE MINING EXPERIMENT Yaye Awa Ba, Philippe. Salomé, Michel. Caillat (LERMA) with credits to : L. Loria, N. Kasradze Archive and Data Mining Goals (i) Search by products not by instrumental configuration
More informationCompact continuum source finding for next generation radio surveys
Mon. Not. R. Astron. Soc. 422, 1812 1824 (2012) doi:10.1111/j.1365-2966.2012.20768.x Compact continuum source finding for next generation radio surveys P. J. Hancock, 1,2 T. Murphy, 1,2,3 B. M. Gaensler,
More informationData Compression for HERA
Data Compression for HERA Paul La Plante, James Aguirre, Bryna Hazelton, Miguel Morales 1. INTRODUCTION The size of the dataset for HERA will soon become intractable for either storage or computing if
More informationEVLA Memo #132 Report on the findings of the CASA Terabyte Initiative: Single-node tests
EVLA Memo #132 Report on the findings of the CASA Terabyte Initiative: Single-node tests S. Bhatnagar NRAO, Socorro May 18, 2009 Abstract This note reports on the findings of the Terabyte-Initiative of
More informationThe Argus 2002 Architecture
1 The Argus 2002 Architecture Steven W. Ellingson 1 1 The Ohio State University ElectroScience Laboratory, 1320 Kinnear Road, Columbus OH 43212. E-mail: ellingson.1@osu.edu 2 I. Acronyms ANP Argus Narrowband
More informationData Provenance and Management in Radio Astronomy: A Stream Computing Approach
Data Provenance and Management in Radio Astronomy: A Stream Computing Approach Mahmoud S. Mahmoud, Andrew Ensor, Alain Biem, Bruce Elmegreen and Sergei Gulyaev Abstract New approaches for data provenance
More informationTHE SKA SCIENCE AND TECHNICAL OPERATIONS PLAN
THE SKA SCIENCE AND TECHNICAL OPERATIONS PLAN Document number...wp2-001.010.010-pla-001 Revision... A Author... SKA Operations Working Group Date... 2010-01-29 Status... Approved for release Name Designation
More informationLBA evlbi Developments. Chris Phillips evlbi Project Scientist 14 September 2007
LBA evlbi Developments Chris Phillips evlbi Project Scientist 14 September 2007 Summary First science paper Offering via normal call for proposals Partner in EXPReS Fringes to Sh as part of APAN Also Da,
More informationNational Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology WISE Archive.
Bruce Berriman / Steve Groom Infrared Science Archive (IRSA), IPAC/Caltech GBB/SLG - 1 WSDC Functional Block Diagram White Sands JPL UCLA HRP H/K MOS Maneuvers SOC Science Team Images FTP Site Ancillary
More informationSun Lustre Storage System Simplifying and Accelerating Lustre Deployments
Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments Torben Kling-Petersen, PhD Presenter s Name Principle Field Title andengineer Division HPC &Cloud LoB SunComputing Microsystems
More informationPDR.01 SDP Architecture
PDR.01 SDP Architecture Document number.. SKA-TEL-SDP-0000013 Context.. ARCH Revision...2 Author. Nikolic Release Date 2015-02-09 Document Classification Status. Draft Release Date: 2015-02-09 Page 1 of
More informationThe CMS Computing Model
The CMS Computing Model Dorian Kcira California Institute of Technology SuperComputing 2009 November 14-20 2009, Portland, OR CERN s Large Hadron Collider 5000+ Physicists/Engineers 300+ Institutes 70+
More informationThe Data exacell DXC. J. Ray Scott DXC PI May 17, 2016
The Data exacell DXC J. Ray Scott DXC PI May 17, 2016 DXC Leadership Mike Levine Co-Scientific Director Co-PI Nick Nystrom Senior Director of Research Co-PI Ralph Roskies Co-Scientific Director Co-PI Robin
More informationLessons learnt from implementing mosaicing and faceting in ASKAPsoft. Max Voronkov & Tim Cornwell ASKAP team 2nd April 2009
Lessons learnt from implementing mosaicing and faceting in ASKAPsoft Max Voronkov & Tim Cornwell ASKAP team 2nd April 2009 Outline - Imaging software ASKAPsoft re-uses LOFAR design Imaging is treated as
More informationHPC NETWORKING IN THE REAL WORLD
15 th ANNUAL WORKSHOP 2019 HPC NETWORKING IN THE REAL WORLD Jesse Martinez Los Alamos National Laboratory March 19 th, 2019 [ LOGO HERE ] LA-UR-19-22146 ABSTRACT Introduction to LANL High Speed Networking
More informationRadio astronomy data reduction at the Institute of Radio Astronomy
Mem. S.A.It. Suppl. Vol. 13, 79 c SAIt 2009 Memorie della Supplementi Radio astronomy data reduction at the Institute of Radio Astronomy J.S. Morgan INAF Istituto di Radioastronomia, Via P. Gobetti, 101
More informationData Management. Parallel Filesystems. Dr David Henty HPC Training and Support
Data Management Dr David Henty HPC Training and Support d.henty@epcc.ed.ac.uk +44 131 650 5960 Overview Lecture will cover Why is IO difficult Why is parallel IO even worse Lustre GPFS Performance on ARCHER
More informationShared Risk Observing
Shared Risk Observing EVLA Advisory Committee Meeting, March 19-20, 2009 Claire Chandler Deputy AD for Science, NM Ops Motivation Early science: Want to get EVLA science capabilities i into the hands of
More informationKVN. Duk-Gyoo Roh, and staffs of KASI & NAOJ + Kagoshima Univ. 8 th VERA User s Meeting, Mitaka NAOJ
KVN Duk-Gyoo Roh, and staffs of KASI & NAOJ + Kagoshima Univ. 8 th VERA User s Meeting, 2010. 09. 16-17. Mitaka NAOJ Contents EAVN & KJJVC project KJJVC framework and status Playback Systems RVDB System
More informationImplementing a Hierarchical Storage Management system in a large-scale Lustre and HPSS environment
Implementing a Hierarchical Storage Management system in a large-scale Lustre and HPSS environment Brett Bode, Michelle Butler, Sean Stevens, Jim Glasgow National Center for Supercomputing Applications/University
More informationControlling Field-of-View of Radio Arrays using Weighting Functions
Controlling Field-of-View of Radio Arrays using Weighting Functions MIT Haystack FOV Group: Lynn D. Matthews,Colin Lonsdale, Roger Cappallo, Sheperd Doeleman, Divya Oberoi, Vincent Fish Fulfilling scientific
More informationEnergy-Efficient Data Transfers in Radio Astronomy with Software UDP RDMA Third Workshop on Innovating the Network for Data-Intensive Science, INDIS16
Energy-Efficient Data Transfers in Radio Astronomy with Software UDP RDMA Third Workshop on Innovating the Network for Data-Intensive Science, INDIS16 Przemek Lenkiewicz, Researcher@IBM Netherlands Bernard
More informationCorrelator Field-of-View Shaping
Correlator Field-of-View Shaping Colin Lonsdale Shep Doeleman Vincent Fish Divya Oberoi Lynn Matthews Roger Cappallo Dillon Foight MIT Haystack Observatory Context SKA specifications extremely challenging
More informationMWA Correlator Status. Cambridge MWA Meeting Roger Cappallo
MWA Correlator Status Cambridge MWA Meeting Roger Cappallo 2007.6.25 Correlator Requirements Complex cross-multiply and accumulate data from 524800 signal pairs Each pair comprises 3072 channels with 10
More informationData Analytics and Storage System (DASS) Mixing POSIX and Hadoop Architectures. 13 November 2016
National Aeronautics and Space Administration Data Analytics and Storage System (DASS) Mixing POSIX and Hadoop Architectures 13 November 2016 Carrie Spear (carrie.e.spear@nasa.gov) HPC Architect/Contractor
More informationNSF s National Radio Astronomy Observatory (NRAO) and CI. Brian Glendenning Head, NRAO Data Management and Software
NSF s National Radio Astronomy Observatory (NRAO) and CI Brian Glendenning Head, NRAO Data Management and Software 1 Campus Cyberinfrastructure, Oct 3 2018 Come Visit Us! The Very Large Array (VLA) is
More informationVLA data reduction part 1:
VLA data reduction part 1: Post-observing, pre-calibration Loránt Sjouwerman, NRAO Outline After the observations: Obtaining your data from the archive Which CPU processes the data? (Home or NRAO) Examine
More informationTGCC OVERVIEW. 13 février 2014 CEA 10 AVRIL 2012 PAGE 1
STORAGE @ TGCC OVERVIEW CEA 10 AVRIL 2012 PAGE 1 CONTEXT Data-Centric Architecture Centralized storage, accessible from every TGCC s compute machines Make cross-platform data sharing possible Mutualized
More informationCase Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy
Case Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy Outline Motivation / Overview Participants / Industry Partners Documentation Architecture Current Status and Services
More informationData Acquisition. Amedeo Perazzo. SLAC, June 9 th 2009 FAC Review. Photon Controls and Data Systems (PCDS) Group. Amedeo Perazzo
Data Acquisition Photon Controls and Data Systems (PCDS) Group SLAC, June 9 th 2009 FAC Review 1 Data System Architecture Detector specific Photon Control Data Systems (PCDS) L1: Acquisition Beam Line
More informationFocus Session on Multi-dimensional Data
Focus Session on Multi-dimensional Data Introduction Mark Allen, Joe Lazio IVOA Interoperability Meeting, ESAC, Madrid, May 20, 2014 CoSADIE Project Science Priority Areas Multi-dimensional Data image:
More informationARCHER/RDF Overview. How do they fit together? Andy Turner, EPCC
ARCHER/RDF Overview How do they fit together? Andy Turner, EPCC a.turner@epcc.ed.ac.uk www.epcc.ed.ac.uk www.archer.ac.uk Outline ARCHER/RDF Layout Available file systems Compute resources ARCHER Compute
More informationAn ESS implementation in a Tier 1 HPC Centre
An ESS implementation in a Tier 1 HPC Centre Maximising Performance - the NeSI Experience José Higino (NeSI Platforms and NIWA, HPC Systems Engineer) Outline What is NeSI? The National Platforms Framework
More informationProgress towards an Asia-Pacific Regional Centre Network. Peter Quinn (ICRAR)
Progress towards an Asia-Pacific Regional Centre Network Peter Quinn (ICRAR) What observatories do Regional Centre? Regional Centre Regional Centre? 2 Data rates and volumes ALMA Science Data Products
More informationHigh dynamic range imaging, computing & I/O load
High dynamic range imaging, computing & I/O load RMS ~15µJy/beam RMS ~1µJy/beam S. Bhatnagar NRAO, Socorro Parameterized Measurement Equation Generalized Measurement Equation Obs [ S M V ij = J ij, t W
More informationData Centres in the Virtual Observatory Age
Data Centres in the Virtual Observatory Age David Schade Canadian Astronomy Data Centre A few things I ve learned in the past two days There exist serious efforts at Long-Term Data Preservation Alliance
More informationArgus Radio Telescope Architecture
Argus Radio Telescope Architecture Douglas Needham http://cinnion.ka8zrt.com http://www.naapo.org Argus Architecture p.1/15 Introduction: Traditional Telescopes Radio telescopes commonly consist of a single
More informationSelf-calibration: about the implementation in GILDAS. Vincent Piétu IRAM. IRAM millimeter interferometry summerschool
Self-calibration: about the implementation in GILDAS Vincent Piétu IRAM 1 About an interferometer sensitivity One usually considers only the noise equation to assess the feasibility of an observation.
More informationDVS, GPFS and External Lustre at NERSC How It s Working on Hopper. Tina Butler, Rei Chi Lee, Gregory Butler 05/25/11 CUG 2011
DVS, GPFS and External Lustre at NERSC How It s Working on Hopper Tina Butler, Rei Chi Lee, Gregory Butler 05/25/11 CUG 2011 1 NERSC is the Primary Computing Center for DOE Office of Science NERSC serves
More informationThe Murchison Widefield Array Correlator
The Murchison Widefield Array Correlator The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation Ord, S. M., B. Crosse, D. Emrich,
More informationAPAN Global Collaboration Linking the World with Light
APAN Global Collaboration Linking the World with Light 1 Andrew Howard Coordinating Engineer Advanced Communications Services and International Development Massive increase in International connectivity
More informationdan.fay@microsoft.com Scientific Data Intensive Computing Workshop 2004 Visualizing and Experiencing E 3 Data + Information: Provide a unique experience to reduce time to insight and knowledge through
More informationSky-domain algorithms to reconstruct spatial, spectral and time-variable structure of the sky-brightness distribution
Sky-domain algorithms to reconstruct spatial, spectral and time-variable structure of the sky-brightness distribution Urvashi Rau National Radio Astronomy Observatory Socorro, NM, USA Outline : - Overview
More informationConcept Design of a Software Correlator for future ALMA. Jongsoo Kim Korea Astronomy and Space Science Institute
Concept Design of a Software Correlator for future ALMA Jongsoo Kim Korea Astronomy and Space Science Institute Technologies for Correlators ASIC (Application-Specific Integrated Circuit) e.g, ALMA 64-antenna
More information