FAULT-TOLERANT COMPUTER FOR THE AUTOMATED TRANSFER VEHICLE

Transcription

1 FAULT-TOLERANT COMPUTER FOR THE AUTOMATED TRANSFER VEHICLE R. Roques, A.Corrégé, C. Boléat Matra Marconi Space, 31, Avenue des cosmonautes, Toulouse Cedex 4, France, telefax: +33 (0) , Abstract Matra Marconi Space is developing their fourth generation fault-tolerant majority voting computers for use on future spacecraft such as the European Automated Transfer Vehicle, a servicing vehicle for the International Space Station. This paper presents how the computer participates in the spacecraft safety concept with emphasis on the critical on-orbit rendezvous phase. The functional architecture of the computer pool is described addressing fault containment layers, inter-channel synchronisation, communication, failed channel detection and passivation. Lastly, the hardware and software design of the test bench, built up to validate this new computer pool, is presented together with some performance measurements. 1. Introduction and fault-tolerance concept 1.1 Objectives The objective of this paper is to summarise the result of the initial development phase of the new generation of Matra Marconi Space fault-tolerant computers including activities focused on defining a concept suitable to the peculiarities of the Automated Transfer Vehicle and the research of key technologies required for the future units to be used by the spacecraft development programme. 1.2 ATV constraints In ATV, the data management system is highly centralised and a unique computer handles all control tasks. This approach requires the centralised data management function to fully ensure the safety of the docking phase to the Space Station without reliance on any other on-board intelligence, since the spacecraft is unmanned, thus leading to the highest level of autonomy as far as safety monitoring and safing actions are concerned Tolerance to hardware faults The fact that we deal with a vehicle with guidance, navigation and control capabilities places tight constraints on real-time performance. The central processing functions have therefore been implemented as a fault masking system, i.e. a majority voting quadruplex computer pool. This means that fault correction can be performed without delaying critical computer outputs, avoiding the delays that are associated with designs that must sequentially detect a failure, passivate it and then perform recovery actions. It also assures that incorrect outputs never occur Tolerance to software faults This is valid, however, only for hardware faults, i.e. those faults which affect only one chain at a time within the computer pool. Software faults could still induce a common mode failure in all units simultaneously with resultant loss of mission. Due to the catastrophic effect on Space Station crew safety which such a software fault could create, should it happen during the rendezvous phase, a solution had to be found to meet safety constraints while avoiding a multiplicity of computer hardware on-board Safety concept Therefore, the computer pool is split up into two zones as shown on figure The higher criticality zone, named the Vital layer is in charge of maintaining the computer pool synchronisation and of monitoring some ATV critical parameters such as attitude and distance rate elements from the docking port. The lower criticality

2 zone, named Nominal Layer is in charge of performing the nominal mission with full performance. NOMINAL LAYER Nominal LN 1 Recovery NOMINAL LAYER VITAL LAYER CHANNEL 1 CHANNEL 2 CHANNEL 3 CHANNEL 4 NOMINAL NOMINAL NOMINAL NOMINAL PROCESSOR PROCESSOR PROCESSOR PROCESSOR VITAL VITAL VITAL VITAL PROCESSOR PROCESSOR PROCESSOR PROCESSOR VITAL LAYER Warm Context (if necessary) Monitoring Contingency Vital Layer Fault Management Vital application sub-layer Input/Output sub-layer Nominal mission Fault detection Short-term Safety Mission continuation with degraded perf. MIL-STD-1553B BUSES SENSORS & ACTUATORS Figure Vital/Nominal breakdown Monitoring function The Vital Layer monitors the system in order to detect the occurrence of wrong behaviour induced by Nominal Layer failures including those that can be caused by software design faults. Both layers use the same sensors in order to limit the amount of hardware on-board, but the monitoring function uses raw sensor data whereas nominal processing uses calibrated/pre-processed data that is produced by the sensor software. The monitoring function is therefore independent from nominal sensorinternal processing algorithmic faults. Collision avoidance and safe mode When a fault is detected, the Vital Layer halts Nominal Layer operations and triggers execution of Contingency software to perform a rendezvous abort manoeuvre that avoids collision with the Station. Once short term safety is ensured, the Nominal Layer can then be reactivated to place the vehicle into a safe mode. This allows troubleshooting operations to be performed from the Ground before attempting a rendezvous retry (Recovery software). The figure summarises which software entities are executed by the pool depending on the mission phase and how they are allocated to Vital Layer or to Nominal Layer. Figure Allocation of software entities 1.3 Fault-tolerance design drivers The implementation of fault tolerance takes advantage ofdevelopments conducted in previous projects in particular in the Hermes space plane technology programme for the overall scientific and algorithmic background and also for the way to handle software faults (in [1]). It also follows the general model described in [4]. Tolerance to two hardware channel faults There are four tightly synchronised channels tied together by broadcast links with checksumed data and supporting double round exchange mechanism This is able to meet a double failure coverage requirement: At the first channel failure occurrence (four computers running, one faulty), all types of faults are covered (arbitrary fault) At the second failure occurrence (three computers active with one faulty, the fourth one passivated after the first failure), all faults which are detectable by the checksum mechanism are covered ( semi-consistent faults as defined in [4]) Tolerance to single application software fault Although located in the same physical channel unit, Vital layer and Nominal layer are segregated to prevent the propagation of software errors from one layer to the other. This is accomplished by allocating the layers to two distinct processors with firewall mechanisms. The Vital layer monitors the Layer and when at least two channels diagnose an AL failure, all AL in the pool are passivated and the FTC enters the Contingency

3 mode. This requires that the failure rate of the Vital Layer software is far smaller than the AL one, which is assumed to be achieved by applying strong design and development constraints (see [3] for use of formal description techniques) and by ensuring that the software size is relatively small to make testing easier. 2. Fault-tolerance implementation NOMINAL VITAL PROCESSOR FDIR MONITORING CONTINGENCY CPU + MEMORY NOMINAL PROCESSOR CPU + MEMORY INTERFACE MEMORY INTER CHANNEL COMM DECOUPLING TO AVOID APPLICATION FAULT PROPAGATION INTER CHANNEL LINK 2.1 General The FTC fault-tolerance implementation can be characterised with respect to: fault containment layers and sub-layers, inter-channel synchronisation, time determinism, fault passivation and reconfiguration 2.2 Fault-containment approach Nominal/vital segregation Vital Layer / Nominal Layer segregation is both physical and functional (see figure ). The two layers are implemented by two distinct sets of electronic boards : the Vital Processor (and 1553B avionics bus Input/Output Board) and the Processor. Memory is also functionally segregated between these layers. The Processor, whatever its fault condition, cannot access or alter Vital layer memory resources since it can only write in a memory space dedicated to Layer/Nominal Layer communications. Written data is checked by Vital layer software for consistency. processor faults can be of two types: faults detected by Vital layer Input/Output software and faults detected by Vital layer application monitoring software. The first group of faults includes hardware and software faults which lead to absence of communication between the two layers or to inconsistent communications. The former can be detected by a kind of watchdog system; the latter will be detected by syntactic/semantic checking (e.g. errors in the layer/vital layer protocol). The second group of faults pertain to those faults which can be detected only at the System level by the application monitoring software. This looks for the compliance during the mission to basic success criteria computed on the basis of sensor data directly acquired from the buses. ACQUIRED DATA VOTED COMMANDS Figure Nominal / Vital physical Segregation Intra-vital segregation Inside the Vital Layer, segregation is provided between the Input/output software and the vital application monitoring and contingency software. The main aim of such a segregation is to make system validation easier by being able to test and validate the two software entities separately. Indeed, these two software products are also separated in the ATV system development cycle : while the Input/Output software is part of FTC development, the monitoring/contingency software is part of the software development process. The introduction of segregation via two integrity levels therefore reduces the risk of introducing "common mode" faults during system integration and allows the two products to be validated independently. Practically speaking, the segregation is provided via special mechanisms (see [2]) and both functions are executed within the same processor. These are : memory segments provided by the microprocessor chipset with each software package running in a dedicated memory area with its own private data a secure communication mechanism between vital application and Input/Output software thus preventing lower integrity application data from contaminating higher integrity data cyclic time-slotted processing resource allocation to prevent processor hogging. 2.3 Inter-channel synchronisation approach The synchronisation between computers is mandatory to ensure the simultaneity of sensor data acquisition, as for

4 processing within each computer and emission of commands towards the actuators. This simplifies error detection in the computer pool and the voting mechanisms implemented in the actuators. The larger the drift in the time base between computers, the greater the processor resource that is lost in tolerating this drift. To meet the mission constraints, the four computers need to be synchronised with an accuracy better than 5 µs. This synchronisation is achieved by a cyclic exchange of a special synchronisation event on the inter-channel link. This exchange is handled by hardware mechanism on software request. With the intrinsic clock stability (20 ppm), a 100 ms synchronisation period is sufficient to maintain a less than 5 µs skew. At a fixed point in each cycle, each computer sends a synchronisation event to the 3 other computers and receives 3 synchronisation events. The difference between the time stamp of its own emission and the time stamp of reception of synchronisation events from the other computers is used to adjust the duration of the next cycle. The choice of the right clock inside the validation window is made according to an algorithm that depends on the configuration of the fault tolerant computer system. This algorithm is performed by software in order to provide flexibility in adapting to reduced configurations (3 or 2 computers) and to required mode changes (pool initialisation, i.e. synchronisation searching or pool nominal operations, i.e. synchronisation keeping ). 2.4 Time determinism Timewise, a robust and deterministic implementation has been selected with cyclic scheduling of Vital Layer tasks (those from the Input/Output software itself but also those belonging to the Monitoring/Contingency software) and cyclic handling of all data exchanges with the avionics buses and with the other channels of the computer pool. This strict determinism policy relies on a hierarchircal time slot concept. At the lowest level, all data exchanges are scheduled within slots. A slot is an atomic period of time which can be used to move and process one message of up to the maximum length of the MIL-STD-1553B avionics bus message. Slots are then gathered to build-up a frame. The frame period corresponds to the Vital Layer cyclic software processing rate to configure data exchanges to be performed by the Vital Layer intelligent controller devices. Lastly, the mission cycle is the complete repetition frequency of all Vital Layer data exchanges 2.5 Fault reconfiguration approach A key function of the computer pool is to manage its configuration in case of failures. Fault detection is achieved on the basis of data generated locally in a channel such as on-line built-in-tests or on the basis of what the other channels "think" of each other, e.g. via the availability or non-availability of valid information on the inter-computer link (ICL) Channel passivation after failure Assuming that a channel failure is detected, the failed channel must be passivated, i.e. its interfaces must be set to such a mode that whatever the nature of the channel failure, further error propagation outside the channel is prevented. Since the nature of the failure is unknown and might obviously affect the ICL or the Vital Layer processing unit, using the inter-channel communication resources (ICL) to passivate a channel is not possible. In fact, the local resources needed for passivation of a channel suspected by the others to have failed must be reduced down to an absolute minimum level. To this purpose, the proposed ATV FTC features a separate hardware consolidation function which consolidates by majority voting the passivation requests coming from other nodes via dedicated hardwired links. This function directly outputs the passivation signals which halt all necessary computer functions and perform interface inhibition. Two dedicated hardwired networks link the channels together: A local error notification network by which a channel tells the others that it has identified its own failure A remote error diagnosis network which allows a channel to tell that is has diagnosed the failure of another channel passivation after software failure A similar approach is implemented to handle Processor reconfigurations. In case of an application software failure, i.e. an abnormal behaviour detected by the Monitoring software running in the Vital layer, the activities in all active Processors are stopped. The Vital Layer then executes contingency software to ensure short-term safety and may ultimately order, if necessary, the reload of the Processors memory. The AP passivation decision (i.e. actuation of the contingency software) results from a voted decision of all

5 channels (at Vital Layer level), using the same mechanism as the computer channel passivation. When two or more application switch requests are received by a channel during the same time window, an interrupt is raised to the fault management software which will then halt the processor software and activate contingency software. Using a dedicated set of hardwired links has been selected for switch performance reasons, since in this case, the switch order are transmitted and hardwire voted in a single shot. The switch delay is therefore negligible compared to the time required by the Monitoring software to detect an out-of-limit condition. Using the ICL for this purpose would require scheduling of the associated data exchange for the next intercomputer link frame, followed by a double round data exchange for voting. 2.6 Hardware/software allocation Each channel is made up of several processing resources allocated to the various functional blocks: application software, vital software, inter-channel communication, avionics bus input data acquisition and command emission, and output data consolidation. Repetitive, processing intensive tasks for intra and interchannel communications and for avionics bus interfaces have been allocated to hardware whenever possible to Off-load the Vital Layer processing and increase the processing capability available to Vital applications, i.e. monitoring/contingency software Limit the period of the software real-time cycle (80 Hz) to allow deterministic software development at low technical risk. The slot level (250 µs) is autonomously performed by hardware and frame and cycle levels are managed by software Reduce the software size, and therefore make its testing easier As a consequence, off-the-shelf intelligent avionics bus interface controllers have been selected and a powerful Inter-Computer Communication Controller integrated device has been designed. This handles cyclic data transfer using tables that are downloaded by the Vital layer processing unit. It handles all communications between the following entities: avionics bus interfaces, inter-computer link, application Layer interface memory and Vital Layer software The synchronisation algorithm is also strongly supported by hardware. The Inter-Unit Communication Controller supports synchronisation message recognition and timestamping on the inter-computer link and automatically generates frame interrupts. The software is only in charge of tuning the synchronisation periodically (100 milliseconds) by adjusting the length of the last slot. 3. FTC development model 3.1 FTC hardware implementation The cmputer pool satisafies on-board space tight hardware budget constraints: small volume, low electrical power consumption, reduced number of electronic boards. Hardware architecture must also be sufficiently modular (standard backplane bus) to accommodate inevitable technical changes and component obsolescence problems that will occur during the ATV programme life cycle of almost 15 years Computer internal design Single processor type A single type of processor, a SPARC ERC32, is used throughout each computer channel for both the Processor (AP) as well as Vital Processor (VP). This allows reuse of software and hardware while minimising the implementation risks. The Processor is the Common Processor Board designed and qualified for DMS-R and Columbus (other European contributions to the Space Station). Data transfer implemented by a dedicated hardware Only limited processing power is needed from the Vital Processor : bitwise input and/or output voting, as well as an Interactive Consistency Mechanism for "Byzantine" voting) are performed in hardware with no performance impact on processor load. This also allows extremely short transit latencies which benefits the application software performance. Modular and upgradable design This selection of a full SPARC architecture will make porting of the software (coded in Ada language) easier should a computer upgrade be needed in a future development of the programme or in another programme. For the same reasons, the internal computer architecture is very modular: boards and backplane comply with the industrial VME standard and the key integrated devices (Inter-Computer Communication integrated circuit, Reconfiguration Unit) do not depend on a particular processor design.

6 The hardware architecture of an FTC channel is depicted on figure V DC POWER SUPPLY UNIT VME SPARE SLOT VME BACKPLANE 1553 I/O RECONF UNIT LOCAL BUS I/F APPLICATION PROCESSOR ERC32 CORE EEPROM VITAL PROCESSOR ERC32 CORE RAM EEPROM OBC93 LOCAL BUS RAM VME MasterI/F VME Slave I/F SHARED MEM Figure FTC channel architecture Inter-computer link design ICC NOMINAL LAYER VITAL LAYER INTER COMPUTER LINK The synchronous and deterministic data exchange mechanism provides more than 5 Mbps effective data transfer capability while including an integrated clock synchronisation mechanism The broadcast transfer topology with an additional message signature (checksum) minimises the risk of Byzantine errors. Message retransmissions which are required by the double exchange mechanism aimed at covering Byzantine faults are performed within the same data transfer slot. The length of inter-computer link messages is 32-data words. This facilitates direct transfer of full 1553B data acquisition messages. From the electrical standpoint, the RS-422 standard selfclocking interface is used. This provides scalable speed/distance properties for various missions 3.4 Test results Data throughput Several data throughput tests have been performed on the test bench. For a test case where each channel acquires on its local avionics buses a 1553B data throughput of 560 kbps (approximately the maximum bus data load) the induced processing load on the Vital Layer processor is about 10 percent, thus leaving most of the processing resource to the Vital application software (monitoring). It is worth pointing out that the same figure is obtained with or without the double round exchange mechanism of the Byzantine faults coverage algorithm. This is due to the fact that this function is fully performed by hardware Synchronisation Tests have been performed at up to a 50 Hz synchronisation frequency. The measured inter-channel clock skew was about 500 nanoseconds. For ATV purpose, a 100 millisecond (10 Hz) synchronisation frequency is selected leading to a skew within the specified 5 microseconds. 4. Conclusion and perspectives This new generation fault-tolerant computer combines a high performance, compact/low cost design and tolerance to application software faults. Integrating software fault tolerance is a significant cost saving factor since, with a classical fault-tolerant computer, vital monitoring needs to be implemented by another set of computers. The activities performed up to now have lead to a technically mature product. It is fully suitable to ATV. Its open architecture and performance margins (in particular short reconfiguration delays) make it adaptable to the needs of future reusable launch vehicles. Acknowledgement: many thanks to the Saab Ericsson Space hardware team (Mrs Svenningsson, MM. Hult, Törnberg, Jernquist), for their contribution to FTC development References: [1] Guidal, David - Development of the Hermes Fault-Tolerant Computer System, FTCS-23, November 93 [2] Roques, Corrégé - A Compact, High Performance Fault- Tolerant Computer Pool Supporting Software Fault-Tolerance, DASIA-96, Rome, May 1996 [3] Ayache, Conquet, Humbert et alii - Formal Methods for the Validation of Fault-Tolerance in Autonomous Spacecraft, FTCS-26, June 1996 [4] Powell - Generic Upgradable Architecture for Real-time Dependable Systems (GUARDS) - Preliminary Definition of the GUARDS architecture - LAAS/CNRS January 1997 Acronyms: AL AP ATV DMS-R FTC ICL VP Layer Processor Automated Transfer Vehicle Data Management System - Russian Service Module Fault-Tolerant Computer Inter-Computer Link Vital Processor