A.J. Faulkner K. Zarb-Adami

Transcription

1 AJ Faulkner K Zarb-Adami March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

2 SKA1-low requirements (after RBS) Frequency: 50MHz 350MHz Scan angle: >45 Bandwidth: 300MHz # of beams: >5 Sensitivity : 500m 2 /K ( MHz) Polarisation: Beam size: Stations: 512 Dual (of good quality) >5 (no beam stitching) Configuration: 50% <600m radius (TBC) 75% <1km radius 95% <3km radius 3 spiral arms of 40km Data rate: ~75Tb/s (total) This is real! March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

3 Some definitions Antenna Min ant spacing: 15m A Station 256 antennas (beams sent for processing) 35m dia Low Frequency AA: 512 Stations Antennas in a random pattern March 2015 LFAA LMC - Trieste A Tile 16 antennas (processed in one module) Andrew Faulkner Kris Zarb-Adami

4 Distance 45km max Station 2 Station Station 1 RPF x Clock Power RPF 2 Clock Power RPF 1 Clock Power Digital Fibre Inner Area Digital Fibre 6km Analogue Fibre CPF Beamforming Correlation Maser clock Data out Power Log Periodic antenna Cooling March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

5 LFAA signal path: Antenna & LNA Receiver Signal Processing LNA e/o RFoF o/e Tile Processing module, ADC Spectral filters Tile Beamforming Part Station beamforming e/o Network Data out Antenna structure Antenna Processing Facility March 2015 Tile Processing Module LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

6 Linking to switched network Rack March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

7 LFAA Data Network 128 racks in total High speed switched network: >40Gb/s Ethernet March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

8 AAVS1 Verification System ( ) March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

9 AAVS1 Verification System ( ) Test overall System On MWA site MRO 400 antennas 4 Stations Correlate with MWA Processing in ASKAP building Build in 2015 Test in 2016 March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

10 AAVS1 system layout MRO Main Building Internet I/F 4 Station box Power 1 s LFAA Mgmt system MWA Control System Patch cables Power Splice Box Power Fibre cables 576-core Fibre splicing 3 25 Clock Dist Data Switch Data Switch MCCS MCCS Servers Servers Signal data Recorder (server) Signal data recorder (server) Beams to MWA Correlator Internet I/F Clocks from MWA/Maser March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami

11 With Ascent Software

12 Interface with TM defined by ICD Interface and control s Calculate and distribute beamforming coefficients Calibrate the antennas for phase and amplitude Distribute calibration coefficients Set up routing between s Monitor hardware status Report hardware and system status

13 Component ACS Common Software EPICS v3 TANGO Device Architecture Object Oriented (CORBA) Flat record database Object Oriented (CORBA) Communication Protocol CORBA message middleware Channel Access protocol ZeroMQ Scalability Limited by CORBA Proven scalability with flat architecture User Interfaces No GUI toolkit X11/Motif + Control System Studio Community LGPL, not fully open source Fully open source Fully open source ZeroMQ limited only by network bandwidth Qt/Java/Web toolkits, Sardana+Control System Studio Usage Limited (~10 institutions) Widespread, choice for ASKAP Linux, Windows, OS X Platform Support Linux VxWorks, Linux, Solaris, Windows, OS X Growing steadily (~25 international partners) Drivers ALMA, Custom Custom, Cosylab COTS support Custom, Cosylab COTS support Training Extensive documentation, limited training Extensive documentation, external onsite training available Extensive documentation, training options available, active forums Configuration Database XML files EPICS database TANGO device server database Archiving Database None, data layer access API ChannelArchiver Historical, temporal and snapshot archives Logging API-based logging of data/actions/events IOC level functions to send logs to text file (eg system log file) Tango Logging Service (TLS), console, file, Tango log consumer/viewer, different log levels per device Debugging No dedicated debugger No dedicated debugger No dedicated debugger Error Handling Backtrace to handling device Alarm states forwarded across links, client checks if alarm condition is present CORBA exception handling, Alarms on attributes Security Container-level protection User-level protection User-level protection

14 General and Non-Functional support and maintainability Communication Infrastructure communication middleware design Monitoring and Control Functionality core operation features and tools Total of 144 points Scores 0: absent features 1: partially available/present features 2: fully available features A score of 1 also reflects minor shortcomings of an otherwise present feature Assessment Score ACS EPICS TANGO

15

16 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface Data Flow 40 Gb-E Network Sky Model Mirror LFAA LMC (MCCS) TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

17

18 TANGO Devices Wrapper around BSP Commands and attributes Device States 14 states New states defined as Enums in TANGO device Data Taps Upload of data into Kafka broker

19 Sensor Information Alarms, events, polling through TANGO Array pointing/calibration Set up as sequences of jobs with OODT Workflow Manager Each job executed and monitored by OODT Resource Manager

20 Logging: Console, file, custom TANGO device Levels: Debug < Info < Warn < Error < Fatal < Off Controlled Access User-based, read, write, read/write rules Error Handling Thrown exceptions

21 Workflow Manager Modeling, executing and monitoring groups of one or more Workflow Tasks Resource Manager Sends jobs to resource nodes to be executed Monitors resource nodes Kafka Cluster Publish-subscribe distributed commit log

22 OODT Core Components All Core components implemented as web services XML-RPC used to communicate between components Servers implemented in Java Clients implemented in Java, scripts, Python, PHP and web-apps Service configuration implemented in ASCII and XML files 28-Jan

23 Core Capabilities File Manager does Data Management Tracks all of the stored data, files & metadata Moves data to appropriate locations before and after initiating PGE runs and from staging area to controlled access storage Workflow Manager does Pipeline Processing Automates processing when all run conditions are ready Monitors and logs processing status Resource Manager does Resource Management Allocates processing jobs to computing resources Monitors and logs job & resource status Copies output data to storage locations where space is available Provides the means to monitor resource usage 28-Jan

24 OODT Processing Use Case 28-Jan

25

26 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (1) Workflow Manager initiates calibration workflow Datatap request job is sent to Resource Manager LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, (Workflow, Resource Resource Management, Management) and and KafkaBroker) Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

27 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (2) Datatap Request commands are sent to TANGO TANGO Host sends commands to s LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

28 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (3) s initiate data upload to Kafka broker, which splits data into files across compute nodes LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

29 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (4) Upload is finished TANGO Host alerts Resource Manager that upload is ready, which in turn alerts Workflow Manager LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

30 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (5) Workflow Manager is (in parallel) waiting for calibration period to elapse After time period, calibration job is submitted to Resource Manager LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

31 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (6) Resource Manager submits calibration job to MCCS compute nodes, and wait for completion, informing Workflow Manager The latest data taps are used LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

32 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (7) Workflow Manager instructs Resource Manager to start coefficients upload job Resource Manager communicates task to TANGO LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

33 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (8) TANGO receives coefficients (in Receiver Device) from s LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

34 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (9) TANGO sends coefficients to s Resource Manager is notified that coefficients have been uploaded to s LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

35 Network Path Control Path Astronomical Data Path Kafka Data Path TM-LFAA Interface (10) Workflow Manager is waits for next calibration period to elapse, and restart calibration process LFAA LMC 40 Gb-E Network Sky Model Mirror TANGO DB LMC Master Node/s OODT (Workflow, Resource Management) and Kafka Broker) MCCS 1: TANGO Host MCCS 2: MCCS 3: MCCS 4: MCCS 5: LFAA-CSP Interface

36 LFAA - TM TM <---> LFAA protocol Finalise ICD that is started Define responsibilities of TM and LFAA Define command structure We look foreward to the meeting outcomes! March 2015 LFAA LMC - Trieste Andrew Faulkner Kris Zarb-Adami