STS-107 Case Study: End-to-End IP Space Communication Architecture

Transcription

1 STS-107 Case Study: End-to-End IP Space Communication Architecture David Israel, James Rash - NASA/GSFC Keith Hogie, Ed Criscuolo, Ron Parise, Francis Hallahan - Computer Sciences Corp

2 Overview Mar The CANDOS Mission Space Components Ground Components End-to-End Data Flows Protocols Tests Results Summary GSAW IP Space Communication 2

3 The CANDOS Mission Communication And Navigation Demonstration On Shuttle (CANDOS) Hitchhiker payload on STS-107, January 16, 2003 Part of the FREESTAR (Fast Reaction Experiments Enabling Science, Technology Applications, and Research) payload Low Power Transceiver (LPT) space test LPT is a multi-channel, programmable transceiver Supports Space Network (SN), Ground Network (GN), and GPS modes Built on stackable PC104 form factor boards Initial flight test of transmitter, receiver, and GPS capabilities Internet Protocol (IP) tests in space Test wide range of standard Internet protocols over SN and GN Primary test of Mobile IP protocol for space use Secondary tests of NTP, UDP, MDP, FTP, SCP, SSH,and IP operations Ground station upgrades Initial test of issues for upgrading SN and GN stations to support HDLC framing and Internet protocols on space links Mar GSAW IP Space Communication 3

4 Space Components LPT Programmable Transceiver 2 - S-band transmitters 12 - S/L-band receivers (SN/GN, GPS) 4 Antennas - high & low-gain transmit, low-gain receive, GPS 233 Mhz 686 processor running Red Hat Linux 6.1 PC104 dual sync serial interface between CPU and transceiver RS-422 clock/data Mar GSAW IP Space Communication 4

5 LPT in STS-107 Bay Hi-gain Antenna Lo-gain Antenna LPT Rcv Data Antenna GPS Antenna Mar GSAW IP Space Communication 5

6 Ground Components Existing antennas, transmitters & receivers at White Sands Ground Terminal (WSGT), Second TDRSS Ground Terminal (STGT), Wallops, and Merritt Island Launch Area (MILA) RF equipment at stations connected to router serial ports Electrical interface adaptation and coding done with GRIDs Routers connected to NASA Closed IONET IP backbone Laptops and workstations in control center (Linux, MacOS, Win98) GRID GRID Closed IONET Mar GSAW IP Space Communication 6

7 Space-Ground - IP Interface for Existing RF Equipment Device similar to a commercial satellite modem was needed to connect custom NASA RF interfaces to commercial routers (electrical interface and coding functions needed) Net PDU Frame IP HDLC Framing IP HDLC Framing Commercial Router (bits) R/S Encode Randomize Conv. Encode R/S Decode Derandomize Conv. Decode Ground-station Router Interface Device Bit sync Modulator Demod Transmitter Receiver Upconvert Downconvert Antenna Existing RF Equipment Mar GSAW IP Space Communication 7

8 Mobile Network Connectivity TDRS-Z DomSat TDRS-W TDRS-E 36,000 Km 36,000 Km 36,000 Km Tracking and Data Relay Satellite System (TDRSS) Low Power Transceiver (LPT) in shuttle bay 36,000 Km 36,000 Km 36,000 Km Space Network (SN) 250 Km Wallops VA GSFC WLPS TDRSS Guam WSGT STGT TDRSS White Sands NM Goddard Space Flight Center Greenbelt MD MILA Ground Network (GN) Merritt Island Launch Area FL Mar GSAW IP Space Communication 8

9 End-to-End Data Flows Standard applications use UDP and TCP APIs to communicate between LPT and end user systems Standard operating systems process IP headers LPT network serial driver connects onboard IP stack with RS-422 synchronous serial interface Frame Relay/HDLC frames over RS-422 clock and data lines between serial interface and LPT transceiver RF signal processed at ground and bitstream (Frame Relay/HDLC) fed to router serial port Ground IP packets delivered over NASA Closed IONET Data formats and protocols to the spacecraft were identical to those from the spacecraft LPT looked like any addressable Internet node Mar GSAW IP Space Communication 9

10 End-to-End Data Flow Data packets addressed directly to multiple destinations by onboard processor Multiple ground systems address data directly to spacecraft, no need to specify ground station LPT Processor & Transceiver CPU End User UDP TCP IP CPU Applications Frame Relay HDLC framing RS-232 Async serial CPU (Linux) UDP Sync serial IP TCP Network driver Sync serial RS-422 Frame Relay HDLC framing RS-422 Coding Ground Station Router Sync serial GRID IP IP Network UDP TCP IP Ethernet Ethernet Ethernet Coding Modulation LPT Modulation Xmit/ Rcvr RF Data GPS RF Mar GSAW IP Space Communication 10

11 Data directed to proper processes using standard and user assigned port numbers Status information logged to syslog and other files for later retrieval Different data kept separate from endto-end More automated file management needed for future missions LPT Software Architecture port TCP GPS rcv ttys1 CAM Data rcv caminterface (GEODE) DSP RS-232 DSP control & status 115 Kbps /var/log/messages Processor (Linux) SLIP Lo-gain xmit Hi-gain xmit ESSC0 ESSC1 driver MDP UDPstat RngSafety blindcmd MIP NTP HTTPD Telnet SSHD ttys port UDP port UDP port UDP port UDP port UDP port UDP port 80 - TCP port 23 - TCP port 22 - TCP RS bps HitchHiker Console SLIP Mar GSAW IP Space Communication 11

12 Data Collection Onboard LPT 10 sec. housekeeping status collected onboard and downlinked as real-time telemetry during communication contacts All information timestamped in SYSLOG facility or application logs (NTP, GPS, MIP, BlindCmd, RangeSafety, etc.) Manually compressed and moved log files to download directories Network Monitoring PERL/SNMP program to monitor router interface traffic, data rates, error indicators, and MIP tunnel status Ethernet LAN analyzer to log all packets in/out of control center Control Center Captured LPT housekeeping telemetry, displayed it, and forwarded UDP status packets to multiple systems with LabView graphical status displays Downloaded files collected and stored by pass for later analysis Mar GSAW IP Space Communication 12

13 Experiment Automated IP routing to current ground station antenna Real-time Telemetry Delivery Onboard Clock Synchronization IP Experiments UDP Protocols Mobile IP- Automatically setup IP routing tunnels to multiple stations/antennas (SN and GN) UDP status packets - to monitor status of LPT over two-way and one-way links addressed to multiple destination addresses Network Time Protocol (NTP) - synchronize and maintain onboard clock referenced to ground time servers TCP Protocols Commanding Reliable File Transfer UDP Blind Commanding - Send UDP command packets to LPT over one-way uplink without Mobile IP or a two-way link Multicast Dissemination Protocol (MDP) - perform reliable file transfer over both two-way and one-way communication links Secure Shell (SSH) & Telnet - login and control experiments from multiple locations Secure Copy (SCP) and File Transfer Protocol (FTP) - reliably transfer files to and from LPT during SN and GN two-way contacts Mar GSAW IP Space Communication 13

14 Security Security & Risk Analysis worked out with NASA network security group All data flows supported over the NASA Closed IP network that supports all other NASA missions All equipment configured and run through security scans before connecting to network Mobile IP used authentication between MN-HA, and FA-HA Encrypted login and data transfers using SSH and SCP One-way UDP paths out through firewall to Internet, no incoming Internet traffic allowed Mar GSAW IP Space Communication 14

15 Overall Protocol Results HDLC framing performed well Variable length frames with CRC-16 error check Operates over uncoded & convolutional coded links, various rates ISO standard supported by standard router serial interfaces Works over one-way links Used over space links for over 20 years Multi-protocol Encapsulation over Frame Relay RFC IETF standard Supported by standard routers and Frame Relay equipment Works over one-way links Serial line analyzers and protocol decodes available UDP/IP packets well suited for space use Standard protocols supported by all routers and computers No connection setup, each packet is self identifying and routable Work over one-way links Easy to pass out through one-way firewall paths Worked for one-way blind commanding and status packets Mar GSAW IP Space Communication 15

16 Overall Protocol Results (cont.) Mobile IP performed very well (~50 ms setup + RTT delay) Mobile IP registration set up tunnel as soon as two-way RF link was established (routers advertising every ~12 sec) Mobile IP only required three packets to set up tunnel on marginal links (advertisement up, registration request down, reg. ACK up) MDP file transfer protocol used extensively Supported transfers over one-way and two-way links Allowed starting file downlinks before uplink was fully established Independent of link bandwidth asymmetry and propagation delay NTP functioned but needs more work for high precision NTP did maintain the processor clock and provided accurate time stamps for all system logs and telemetry samples Didn t have good HW for precision timing, simple PC104 computer clock, no thermal control, no 1 PPS source, No independent onboard time reference to measure against SSH and SCP used successfully Required two-way link Mar GSAW IP Space Communication 16

17 UDP Telemetry Packets UDP packet uses in space communication Each packet has a full network source & destination address as well as UDP port number information to further categorize and route data packets UDP unaffected by link delays and data rates Functions properly without needing Mobile IP or two-way link Current NASA data is carried around the ground in with CCSDS/4800BB/TDM in UDP packets LPT real-time telemetry status packets CANDOS used simple tab-delimited ASCII strings to send real-time status data to specific ports on different systems for different types of data Telemetry packet sizes variable from ~ bytes depending on the numerical values in them UDP packets delivered to open Internet via one-way only path through NASA firewall UDP flexibility Different telemetry packets delivered to various destinations based on onboard addressing decisions. Easily built status display programs using PERL, LabView, TREK Mar GSAW IP Space Communication 17

18 Mobile IP Downlink data is routed normally Need to automatically determine which ground station to send commands through (same problem for cars, PDAs, laptops) Mobile device registration with ground agents supports automatic uplink routing configuration Robust protocol worked under non-optimal links, only needed to get 3 packets across the RF link to set up tunnel Cisco IOS Spacecraft address x subnet Control Center x subnet ASPC (Home Agent) Wallops (Foreign Agent) Mobile IP Tunnel MILA (Foreign Agent) x subnet Mar GSAW IP Space Communication 18

19 Mobile IP Performance seconds roundtrip MIP setup (.9 sec TDRSS propagation) LPT 9 8 Registration Request. -MN->FA Reg Accept Rcvd - MN 7 6 Adv Rcvd -MN ~300 ms propagation over TDRSS link ~300 ms propagation over TDRSS link ~300 ms propagation over TDRSS link MIP Advertisement - FA->MN 5 Reg Accept to MN - FA->MN 4 Reg Req.Rcvd -FA FA 3 Reg Req.Processing -FA Reg. Accept Processing - FA Tunnel setup 2 Reg Req.to Home -FA->HA 1 Reg. Accept - HA->FA HA 0 HA Authentication & Processing 13:25: :25: :25: :25: :25: :25: :25: :25: :25: :25: :25: :25:12.5 Mar GSAW IP Space Communication 19

20 10 TDRSS Mobile IP Session LPT Registration Request. -MN->FA 9 Reg Accept Rcvd - MN 8 Adv Rcvd -MN 7 MIP Advertise - FA->MN 6 Enlarged in previous slide Reg Accept to MN - FA->MN 5 FA Reg Req.Rcvd -FA 4 Reg Req.Processing -FA 3 Reg Req.to Home -FA->HA 2 HA Reg Accept - HA->FA 1 HA Authentication & Processing 0 13:20:00 13:23:36 13:27:12 13:30:48 13:34:24 13:38:00 13:41:36 13:45:12 13:48:48 13:52:24 13:56:00 Mar GSAW IP Space Communication 20

21 Network Time Protocol Time Offset (sec) / /21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 2/ Mar GSAW IP Space Communication 21

22 NTP During Jan. 24, Station Access Up/Down Rate (Kbps) Pass Duration (Min.) STGT SA 32/ STGT SA 32/ STGT SA 32/ STGT SA 32/ WSGT SA 32/ STGT SA 32/ Testing different MIP version Two-way link problems STGT SA 16/2,4,8,16 Time Offset (sec) 1/24 00:00 1/24 04:00 1/24 08:00 1/24 12:00 1/24 16:00 1/24 20:00 1/25 00: MILA 2/ WSGT MA 16/16,128 WSGT SA 32, WSGT MA 16/ Mar GSAW IP Space Communication 22

23 Commanding Commanding using Linux shell scripts Shell scripts very useful for lots of last minute updates and fixes, onboard automation Demonstrated using cron onboard to to execute scripts to perform some traditional time-tagged commanding UDP Blind commanding over manual tunnel and automatic MIP tunnel LPT process listening on port, received packet containing name of Bash script to execute and any parameters to pass to script TCP Secure Shell (SSH) login - secure, encrypted login from multiple locations, some sessions with multiple, simultaneous access Interactive onboard flight system maintenance SCP uploads of scripts, and other configuration information (similar to table load and stored command uploads) Telnet used across very slow HitchHiker 1200 baud access link since it had less overhead and was on a dedicated link Mar GSAW IP Space Communication 23

24 File Transfers Data compression Some files were compressed with gzip (some 10 to 1 compression) Others sent uncompressed if time and bandwidth available File transfer applications unaware of compression, they moved files UDP File Transfers MDP over one-way and two-way links, automated hot-directory MDP across multiple ground stations with MIP handovers TCP File Transfers Uploaded new executables, images, data, firmware FTP used for special cases with limited bandwidth links SCP sessions across multiple ground stations with MIP handovers SCP over two-way links with multiple locations- control center, MSFC, OMNI lab (remote users didn t know what the path was) MSFC picked up putty, public domain SCP software, the day before they downloaded files from the LPT to MSFC Mar GSAW IP Space Communication 24

25 Multicast Dissemination Protocol UDP based, reliable file transfer protocol developed over 5 years ago at Naval Research Lab UDP protocol avoids TCP problems with delay and intermittent links and lets application deal with to space link delay and intermittent connectivity issues Multicast protocols developed to primarily send data to thousands of receivers and minimize responses using primarily NACKs and a final ACK On CANDOS MDP was also used over TDRSS MA Returnonly links to deliver data with final NACKs, retransmissions, and ACKs during two-way contact time Work continuing as NACK Oriented Reliable Multicast (NORM) in IETF Reliable Multicast Transport Working Group Mar GSAW IP Space Communication 25

26 1,000, , , , ,000 MDP File Transfer Analysis CANDOS MDP File Downlink Date: 2003 Jan 20 Time: 14:27 GMT File size: Bytes Throttle Rate: 12kbits/sec Fwd Link Rate: 16kbits/sec Rtn Link Rate: 16kbits/sec BER: 1e-6 Path: TDRS-east Groundstation: STGT Pkt #641 Pkt #704 Pkt #769 Pkt #835 Pkt #897 2 Retrans: pkt #691 pkt #696 4 Retrans 1 Retrans 9 Retrans 5 Retrans: pkt #705 pkt #747 pkt #749 pkt #762 pkt #766 End of file Pkt # , ,000 In-sequence transmission of pkt #257 5 Retrans: pkt #576 pkt #617 pkt #621 pkt #631 pkt # , , ,000 0 Beginning of file Dropped Pkt #197 NAK for #197 1 Retrans: pkt #197 NAK for #576 #617 #621 #631 # Packets NAK for #691 #696 NAK for #705 #747 #749 #762 #766 NAK for #777 #782 #789 #790 #794 #799 #806 #809 #815 #831 NAK for #806 NAK for #832 #845 #865 #870 Mar GSAW IP Space Communication 26

27 Anomalies Software issues MosquitoNet MIP timeout handling software was not correct and long RTT with TDZ resulted in excess reregistrations MDP throttle rate slow to change for low rates (fix coming) Operational Issues Manual tunnel and automated tunnel conflict due to operator error Coordination of multiple simultaneous users to avoid bandwidth conflicts, access control, Onboard software configuration management is important with multiple user access When SCP has trouble use MDP, more forgiving when two-way starts fading Need to monitor/manage link utilization, excess UDP data can clog link and cause long delays, also triggered MIP excess reregistration Mar GSAW IP Space Communication 27

28 Operational Enhancements This mission focused on testing protocols and functionality and a long-term operational mission would need more automation Ground network - automated setup and monitoring of manual IP routing tunnel for use in blind commanding Automated onboard file management coordinated with file transfers and RF system Connections between MIP, MDP and onboard transmitter/receiver status for automated operation Automated error handling, space and ground Mobile IP software implementation with better handling of long RTT Possible onboard hardware/software enhancements Full router to do prioritized traffic queuing, security, mobile routing, 1 PPS signal from GPS to drive onboard NTP daemon More stable clock Mar GSAW IP Space Communication 28

29 Summary Successful demonstration of using standard IP protocols and applications for space communication and operations Successful detailed measurement of protocol performance and operation with lots more data to analyze Successful demonstration of long-term, scalable concepts that can be used for command, control, and data collection for a wide range of future systems: satellite science missions balloon and aircraft systems ground sensors collaborative/ad hoc systems with large numbers of devices Internet protocols allowed us to quickly and easily try new applications and operation scenarios during the mission Standard, off-the-shelf products worked well and can work much better with a little design/configuration effort and real flight hardware Mar GSAW IP Space Communication 29

30 Jan. 16, :39:00 GMT Mar GSAW IP Space Communication 30