GLAST Large Area Telescope:

Transcription

1 GLAST Large Area Telescope: Gamma-ray Large Area Space Telescope Flight Software J.J.Russell/A.P.Waite Stanford Linear Accelerator Center

2 Topics To Be Covered Functional Requirements Baseline Architecture Derived Functional Requirements Event Filtering Command Control and Configuration Housekeeping and Low Rate Science Resulting Software Architecture Code Development Software Releases Flight Team Experience Recommendations From January Review J.J.Russell/A.P.Waite LAT Flight Software 2

3 Functional Requirements Instrument Flight Software Interfaces S/C Commanding LAT Control System Startup Functions 1553 Solid State Recorder CPU Reset PPS Interrupt GBM Interrupt Watchdog Off CPU Timers EEProm / File DAQ I/O Authentication Real Time Commands Timed Command Scripts Event Driven Commands GPS Message Attitude Message GBM Message Transient Alert S/C Repointing Request File Upload File Download Save Audit Trail Thermal Mode Control Configuration Control Fault Handling Save Audit Trail Processor Boot Instrument Startup Housekeeping Low Rate Science Calibration Diagnostics Identify Transients Event Monitoring Event Filtering Note: This is instrument flight software, not spacecraft flight software All instrument flight software is updatable during operations J.J.Russell/A.P.Waite LAT Flight Software 3

4 Baseline Architecture DAQ Front End Electronics Event Processing Unit (1 of n) Command Control Configuration Spacecraft Interface Unit Command Control Configuration 1553 Service Spacecraft Discrete Signals 1553 Event Dispatch Calibration CNO Event Filter SIU <-> EPU SIU <-> EPU CFG/HSK/LRS Dispatch Calibration CNO Science Configuration Housekeeping LRS SSR Service SSR J.J.Russell/A.P.Waite LAT Flight Software 4

5 Derived Functional Requirements (SIU) Instrument Flight Software Interfaces S/C Commanding LAT Control System Startup Functions 1553 Authentication Thermal Processor Boot Housekeeping Solid State Recorder Real Time Commands Instrument Startup Low Rate Science CPU Reset Timed Command Scripts Mode Control/Synch PPS Interrupt Event Driven Commands Configuration Control Calibration (master) GBM Interrupt GPS Message Diagnostics (master) Attitude Message Fault Handling Watchdog Identify Transients Off CPU Timers GBM Message Save Audit Trail Event Post Processing EEProm / File Transient Alert S/C Repointing Request DAQ I/O File Upload EPU I/O File Download Save Audit Trail J.J.Russell/A.P.Waite LAT Flight Software 5

6 Derived Functional Requirements (EPU) Instrument Flight Software Interfaces S/C Commanding LAT Control System Startup Functions SIU I/O Mode Control/Synch Processor Boot Calibration (slave) Event Builder I/O Diagnostics (slave) CPU Reset Event Monitoring Off CPU Timers Event Filtering J.J.Russell/A.P.Waite LAT Flight Software 6

7 Event Filtering - Introduction Requirements On & Off-Board Science Sets photon capture efficiency Constraints Processing Rate Orbit max trigger rate is ~10kHz: ~100 sec per event system-wide Downlink Bandwidth Orbit average of ~300kbps: ~ 20 Hz background / 10Hz physics Send few events. Filter must reject ~99.8% of background events Send small events. Design dense event formats. Processing Throughput Is A Complex Function Of Event size (longer events result in more memory accesses) Event layout (memory accesses can be saved with good layout) CPU architecture (CPU speed, memory speed, execution units, ) Carrying two target processors complicates the picture Filtering code Algorithms Implementation J.J.Russell/A.P.Waite LAT Flight Software 7

8 Event Filtering Development Path Using Monte Carlo Events Generated In GLASTsim Event size includes noise at prescribed rates (see next slide) Event layout is near final DAQ format Algorithmic Development Designing and debugging on SUN/LINUX boxes Measuring performance on Motorola MV2303 and RAD750 Event Features Used In Current Round Of Analysis TKR layer hit bits (very fast access) ACD tile hit bits (disordered but access still fast) CAL energy sums (slowest access needs coarse calibration constants) Minimal track finding J.J.Russell/A.P.Waite LAT Flight Software 8

9 Event Filtering Event Size Event Sample Generated With 1 x 10-2 CAL noise occupancy x 1536 x 2 logs ends 1 x 10-4 TKR noise occupancy x 884k strips Subsystem Hits Noise Fixed Overhead Bits/hit Volume Bits Bytes CAL 26 logs 30 logs 32 bits / tower * 16 towers TKR 93 strips 88 strips 72 bits / tower * 16 towers ACD 5 tiles 1 tile TRG Typical Event Monte Carlo Measurement Bytes J.J.Russell/A.P.Waite LAT Flight Software 9

10 Event Filtering CPU Comparisons Feature Comparison Processor Board CPU Instructions Per Clock Speed (MHz) Memory Wait States Execution Units L1 Cache (kbyte) Cycle CPU Memory Integer Float Load/ Save Branch Prog Data Motorola MV NRL custom BAE RAD Performance Comparison Processor Board CPU Memory CPU Motorola MV NRL custom BAE RAD J.J.Russell/A.P.Waite LAT Flight Software 10

11 Event Filtering - Results Cut No CAL LO + Veto Tile ACD Splash Veto (pass 0) CAL < 350Mev + Veto Tile CAL < 10 Mev + Any Tile ACD Splash Veto (pass 1) TKR tower match with ACD top tile TKR tower match with ACD side tile No connection between CAL energy & TKR CAL Energy Layer 0/Total Energy <.01 Events Analyzed (%) Rejected (%) (100.0) 9923 (64.4) 5497 (35.6) 1566 (10.2) 3931 (25.5) 224 (1.5) 3707 (24.0) 464 (3.0) 3243 (21.0) 69 (0.4) 3174 (20.6) 424 (2.7) 2750 (17.8) 304 (2.0) 2446 (15.9) 1152 (7.8) 1294 (8.4) 156 (1.0) <Time> sec CAL Energy Layer 0/Total Energy > (7.4) 94 (0.6) Before track finding 1044 (6.8) (93.2) TKR/ACD matching 1044 (6.8) 262 (1.7) Projects into skirt region 782 (5.1) 83 (0.5) E < 350 Mev, Number of Tracks < (4.5) 461 (3.0) Final 238 (1.5) (98.5) J.J.Russell/A.P.Waite LAT Flight Software 11

12 Compared With Jan PDR Event Filtering - Summary Rejection Rate has gone from 82.6% 98.4% Time (MV2303) has gone from sec sec Still Need To Go From 98.4% 99.8% But have the numbers on our side >95% rejection in 14 sec/event (RAD750) leaves 1.4 msec/event To preserve 100% margin in one CPU, still have 700 sec/event» This is 50 times the event processing time used so far More Confident In Where We Stand Do not need to extrapolate estimates as far Know the target CPU performance much better 1 BAE 750 or 1.5 NRL 603e is sufficient to do filtering with 100% margin Can Now Return To Estimating Other CPU Demands Previously considered small compared to filtering J.J.Russell/A.P.Waite LAT Flight Software 12

13 Command Control and Configuration The SIU Receives Commands On A 1553 Bus In CCSDS Format Commands Belong To One Of Two Types Real time commands that execute immediately Block Commands, used for Executable code uploads Table/parameter uploads Command list uploads Time Ordered LAT Activity Driven By Command List Each command is time tagged to a precision of one second Command execution is scheduled with an accuracy of 10 msec Flight software will store up to 48 hours worth of such commands J.J.Russell/A.P.Waite LAT Flight Software 13

14 Command Control and Configuration SIU Implements LAT Control SIU Receives And Executes Commands From Real time commands Command lists On-board events SIU Controls LAT Configuration By SIU/LAT command protocol to Primary instrument power distribution DAQ electronics boards (GASU/TEM) Via GASU/TEM to CAL/TKR/ACD front end electronics By SIU/EPU communication path to EPUs Scatter/gather operations, synchronization points, etc. SIU Controls LAT Operational Mode Normal operations, engineering mode, safe mode, etc. J.J.Russell/A.P.Waite LAT Flight Software 14

15 Command Control and Configuration A Configuration Completely Describes The State Of The LAT Configurations are used To meet scientific objectives To protect LAT health To control redundancy paths and mitigate component failure Configurations are managed by the SIU The SIU exchanges configurations across two interfaces To and from the ground To and from the DAQ system A static configuration description is > 380kBytes 360 kbytes in TKR front end registers 17 kbytes in CAL front end registers 1 kbyte in ACD front end registers 2 kbytes in TEM/GASU Plan to use a more efficient representation J.J.Russell/A.P.Waite LAT Flight Software 15

16 Housekeeping and Low Rate Science Housekeeping A set of metrics monitoring the health of the LAT Monitored by the SIU Run independently of the event path ( always on ) The 1 Hz housekeeping scans collect Temperatures, voltages, currents, (via SIU/LAT I/O) EPU metrics idle time, memory usage, (via SIU/EPU I/O) Low Rate Science A set of counters which can be multiplexed to test points Controlled and monitored by the SIU Run independently of the event path ( always on ) All counters can be started and stopped synchronously Counters can target a variety of information examples ACD tile singles rates A single tower s three-in-a-row trigger rate J.J.Russell/A.P.Waite LAT Flight Software 16

17 Software Architecture (SIU) Control / Mode Command Queue Finite State Machine Tasks Event Post-processing Calibration Diagnostics Transient Analysis SIU Housekeeping Low Rate Science CNO Analysis LAT Housekeeping SAA Monitoring Services Event Dispatch Command Dispatch LAT Configuration SIU / EPU Scatter / Gather Watchdog Update CCSDS Timers Boot Interrupt (S/C discretes) Drivers SIU / EPU Communications SIU / LAT Communications Watchdog 1553 EEProm / File Solid State Recorder J.J.Russell/A.P.Waite LAT Flight Software 17

18 Software Architecture (EPU) Control / Mode Finite State Machine Tasks Event Filtering Calibration Diagnostics Event Monitoring EPU Housekeeping Services Event Dispatch Command Dispatch SIU / EPU Scatter / Gather Watchdog Update Timers Boot Drivers SIU / EPU Communications EPU / Event Builder Watchdog J.J.Russell/A.P.Waite LAT Flight Software 18

19 Software Code Size Estimates Have Performed A Detailed Lines-Of-Code Estimate Sent to IV & V and available as a handout TARGET Common Code EPU SIU Total Lines of Code 20,000 30,000 50, ,000 Estimate Is Not Complete But reasonable bounds would be K This Is Down From The K Estimate We Gave In Jan PDR That was based on inflating balloon flight code by a factor of 5 This is a bottoms up estimate J.J.Russell/A.P.Waite LAT Flight Software 19

20 Code Development (1) Coding Environment And Tools Decisions Made Target processor is PowerPC class Real time operating system is VxWorks Embedded system development environment is WRS/VxWorks Embedded system development host is Sun/Solaris Coding language is C Special purpose exceptions permitted, for example» C++ wrappers if needed» Perl, Tcl/Tk, etc or similar for scripting Host images built for Sun/Solaris and Linux (only) Decisions Pending Use SCL for flight command, control and configuration? no Reason: Difficult to support on NRL 603 board, memory limitations Action : Examine other solutions for command, control & configuration Use SCL for software test executive? no Reason: Wish to follow EGSE solution (in flux) Action : Wait for EGSE solution, understand FSW consequences J.J.Russell/A.P.Waite LAT Flight Software 20

21 Code Development (2) Code Management Functions Track source code (who edited what, when and why) Define and support a modular software architecture scheme Build object code (for multiple target systems) Define and support a software lifecycle Provide the mechanism to implement release control Provide the mechanism for software distribution Tools adopted/written CVS implements source code tracking and release identification CMT implements source modularization and object building CMX implements lifecycle definition and software distribution Status Built and used for the balloon flight (50k LOC across 5 sites) J.J.Russell/A.P.Waite LAT Flight Software 21

22 Code Development (3) Documentation Project level and FSW top level design/discussion documents Technology: Cyberdocs Status: Implemented Cyberdocs already managing project documents Code descriptive documentation (usage manuals) Technology: PDF documents, version controlled through CVS Basis of as built software documentation Status: A work in progress Code reference documentation (interface/internals definitions) Technology: Doxygen Status: Implemented Documentation auto-published to web by build process J.J.Russell/A.P.Waite LAT Flight Software 22

23 Code Development (4) Continuous Testing Objectives Verify code function, performance and hardware integration Methods White box testing Allowed to look inside code unit to test» Rarely used code paths, boundary conditions, subtle timing effects Black box testing Exercises the code unit interface only Otherwise similar to white box testing Formal Release Testing Objectives Verify compliance with project requirements and interface specifications Methods Develop and execute formal test plan Status Flight Software Test Plan available in draft form Requires the development of a supporting infrastructure J.J.Russell/A.P.Waite LAT Flight Software 23

24 Software Release For Engineering Model One Hardware 1 Partially populated tower 1 Tower Electronics Module 1 COTS CPU (VME) Ethernet Serial port SIU/LAT I/O card (or simulator) Event builder card (or simulator) Software Interfaces (other than VxWorks) SIU/LAT I/O driver Event builder driver Configure all registers CAL TKR TEM Acquire housekeeping data Acquire LRS data Acquire single tower event data Status: Development complete against preproduction electronics Deployed to field in I&T test stands J.J.Russell/A.P.Waite LAT Flight Software 24

25 Software Release For Engineering Model Two Hardware >1 Towers >1 Tower Electronics Modules >0 GASU (or simulation) Includes Event Builder 1 COTS SIU CPU (cpci) Ethernet Serial port SIU/spacecraft I/O card (SIB) SIU/LAT I/O card (LCB) >0 COTS EPU (cpci) Ethernet Serial port SIU/EPU I/O card (LCB) Software SIU EPU Interfaces SIU/LAT I/O driver (multiple Tx/Rx) SIU/EPU I/O driver Configure all registers Acquire HSK (multiple Tx/Rx) Acquire LRS (multiple Tx/Rx) Handle EPU forwarded events Begin command/control Begin mode/synch model Interfaces Event builder driver (multiple Rx) SIU/EPU I/O driver Configure event builder Acquire merged event data Monitor events / forward to SIU J.J.Russell/A.P.Waite LAT Flight Software 25

26 Software Release For Calibration Unit Hardware Many Towers Many TEMs Many hardware simulators >1 GASU Includes Event Builder 1 pre-flight SIU CPU (cpci) Ethernet Serial port SIU/LAT I/O card (SIB) SIU/EPU I/O card (LCB) S/C simulator >1 pre-flight EPU CPU (cpci) Ethernet Serial port SIU/EPU I/O card (LCB) Software SIU all previous plus Interfaces SIU/EPU I/O driver (multiple Tx/Rx) 1553 driver SSR driver Extend command/control Extend mode/synch model Run control Distributed calibration EPU all previous plus Event filtering Complete event monitoring J.J.Russell/A.P.Waite LAT Flight Software 26

27 Flight Team Experience More than 50 man years designing/coding/running embedded systems Balloon flight: successful proto-fsw development and demonstration Items in green are identical between the balloon and flight Processors Multi-processor Environment Real Time Operating System Trigger Rate Capacity (measured/required) Event Filtering Telemetry Uplink Telemetry Downlink Configuration Management Code Distribution Code Documentation (reference) Code Documentation (evolving manuals) Project Documentation Code Test Lines of code Balloon PPC Class Yes VxWorks 5k None (16 bits/sec) CCSDS protocol CMX/CMT/CVS CMX/CVS Doxygen (Ad hoc) CyberDocs (Ad hoc) 50k Flight PPC Class Yes VxWorks 10k ~500:1 CCSDS protocol CCSDS protocol CMX/CMT/CVS CMX/CVS Doxygen Framemaker/Office/CVS CyberDocs Structured K J.J.Russell/A.P.Waite LAT Flight Software 27

28 Progress Related To Jan Review Recommendations Manpower Have two open positions and three candidates New lines of code estimates reduce the size of the project Peer Review Instrument level review process added to management plan Discussions opened with potential external chair (T.Shalk) Documentation The following documents have been created/revised: Document LAT-SS LAT-MD LAT-TD Title LAT Flight Software Specification Level IV LAT Flight Software Management Plan LAT Flight Software Test Plan Status In signature cycle with subsystem leads prior to release. New version in draft. Draft available in Cyberdocs. J.J.Russell/A.P.Waite LAT Flight Software 28

29 Summary Software Functionality Identified New - Detailed package description with size estimates Software Development Methods Identified Many methods already available and tested on balloon flight New - Have extended these to all phases, including BSP dev. New - Have solidified release identification and tracking Software Releases Identified For Each Engineering Stage Progress On Issues Identified By FSW Group In Jan PDR Complete the event filter performance analysis Close to accomplishing this goal (with near production quality code) To do Photon efficiency (physics) Make the code available to the offline (purely mechanical) J.J.Russell/A.P.Waite LAT Flight Software 29

30 Summary (continued) Progress On Issues Identified By FSW Group In Jan PDR (cont) Investigate SCL as the vehicle for flight C 3 Have decided not to use SCL on board Investigate SCL as the vehicle for flight test executive Will follow EGSE lead Develop software manual level documentation guidelines Guidelines outlined in Software Management Plan Develop an automated software test strategy Software Test Plan drafted Gives us a roadmap to build an automated system Considerable Progress Has Been Made Including Boot non-absolute images; ELF stripping, compression 1553 Double ended (BC/RT) test bed in VME J.J.Russell/A.P.Waite LAT Flight Software 30

31 Additional Information

32 T&DF Overall Architecture Tower Electronics Module Interconnect Function Event Processing Unit Spacecraft Tower Front End Electronics TKR CAL Tower PWR Mgt HSK/LRS TKR ASICS CAL ASICS Trigger Primitives C O N T R O L 16 ~4 1 Event Builder Processor CPU I/O P C I B U S ACD Electronics Function Global Trigger Function Spacecraft Interface Unit ACD Front End Electronics ACD Tower PWR Mgt HSK/LRS ACD ASICS Trigger Primitives C O N T R O L C O N T R O L HSK/LRS TRG FIFO TRG Configuration Trigger Logic ~4 DAQ PWR Mgt CPU I/O LAT I/O Processor Spacecraft I/O P C I B U S 1553 SSR Discretes Key Trigger Primitives: Trigger Message: Event Data: All event fragments: Configuration/Housekeeping/Low Rate Physics Data: Command, Configuration and Control: J.J.Russell/A.P.Waite LAT Flight Software 32