SpaceOps 2006 Conference AIAA 2006-5677 : Deployment of Automated Execution for EUMETSAT Ivan Dankiewicz and Roger Thompson SciSys Ltd, Methuen Park, Chippenham, Wiltshire, SN14 0GB, UK e-mail: Ivan.Dankiewicz@scisys.co.uk, tel: +44(0)1249466466 Julian Long SciSysLtd, Methuen Park, Chippenham, Wiltshire, SN14 0GB, UK Automated EXecution () is a generic portable procedure definition and execution tool that enables automation of spacecraft operations and EGSE checkout. Designed to be performant, portable, distributable and scalable, its lightweight distributed architecture means that is able to support any mission configuration, from checkout for a single payload instrument to mission operations for satellite fleets or constellations. This paper describes the deployment of as the replacement automation component of the EUMETSAT Meteosat Second Generation (MSG) mission control system. EUMETSAT s operational experience and expertise in the use of the first generation tool has provided valuable feedback in the development and enhancement of and its integration into the Meteosat Second Generation (MSG) system. has a graphical editor to support the development of operations and test procedures by spacecraft engineers. This allows procedures to be built by dragging-and-dropping steps onto a flowchart view. Facilitating the editing of individual details in procedural steps has been a key objective and the editor has greatly improved functionality and performance compared to its predecessor. Emphasis has also been placed on procedure execution modes that simplify procedure validation. can be deployed in conjunction with other procedure authoring tools by importing standard or custom procedure scripting languages. In the case of EUMETSAT, the European Organisation for the Exploitation of Meteorological Satellites, existing automated procedure definitions were automatically imported into the environment. Nomenclature = Automated EXecution API = Application Program Interface CF = Central Facility ECSS = European Cooperation for Space Standardisation ICOL = Integrated Common Operations Language EUMETSAT = European Organisation for the Exploitation of Meteorological Satellites MSG = Meteosat Second Generation OL = Operations Language PLUTO = Language For Users In Test and Operations POD = Overview Display SPED = Single Execution Display SSM = Space System Model TM = Spacecraft Telemetry UNiT = Universal Intelligent Toolkit T I. Introduction He first section of this document describes the core tool. The second section describes the tailoring and deployment of for the EUMETSAT MSG control system. The final conclusion section describes the lessons learnt in the deployment of in the EUMETSAT MSG control system. 1 Copyright 2006 by SciSys Ltd. Published by the, Inc., with permission.
II. Product Description A. The Automation Model The model of automated operations illustrated in Figure 1 has been used successfully by SciSys for several large projects. The procedure execution product is a key component of this model. The model supports an off-line interactive mission planning layer, plus on-line automated PREDICTED EVENTS/CONTACTS, PLANNING REQ/RSP operations support functions at three levels: Off-line mission planning Scope PLAN GB Activity SCHEDULE Task GB Activity PROCEDURE Thread Step APP. OBJECTS Actions Set Parameter Send Command Raise Event Control Control Schedule OB Activity OB Activity Data Parameter Value Command Status Status Schedule Status Underlying Monitoring & Control System Figure 1: Model of Automated Operations On-line Schedule Execution s implemented by Application Objects implemented by ICOL Schedule Execution layer Execution layer Operations Language layer. implements the procedure execution layer and the Operations Language is implemented by the Integrated Common Operations Language (ICOL). ICOL provides an interface with the underlying M&C system allowing expressions, which reference system data, to be evaluated, and system actions such as sending a command to be executed, asserting a parameter value, or raising an event. The proven conceptual model of SciSys UNiT toolkit for automated operations procedures has been re-used in the context of the environment. Operations language (OL) execution in UNiT was implemented using Gensym s G2 environment; in, Gensym G2 has been replaced by ICOL. The procedure definition format has considered issues of compatibility and interoperability with PLUTO, a procedure language standard currently being specified by ECSS 1. B. The Model of Execution s correspond to pre-defined operational activities that can either be scheduled as a ground-based activity, or manually initiated. s may also call other Sub-s, permitting their decomposition into smaller, more maintainable units which can be reused in several different operations. Primary Thread Trigger (wait & check condition) Single Pass Multi Pass Secondary Threads Confirmation (wait & check condition) Condition on completion. Trigger Phase Outlining Flow Control Execution Phase Step Trigger Step Body Step Confirmation Confirmation Phase Figure 2: Model of Automated Operations A procedure comprises a public interface (including arguments), local variables, and a set of Thread definitions with constituent Steps, as shown in Figure 2. s contain a single Primary Thread and, optionally, a number of Secondary Threads. Each Thread constitutes an independent flow of control through the procedure. Threads comprise an Activation Condition and a sequence of Steps. A thread activates following the occurrence of its Activation Condition and then proceeds to execute the steps in sequence. Threads may be defined to be single-pass or multi-pass. Singlepass threads will execute once only within the context of a procedure invocation. Multi-pass threads can execute repeatedly, resetting and awaiting the next occurrence of the Activation 2
Threads may be controlled by other threads within the same procedure. Threads may be enabled and disabled: However only enabled threads can trigger. Active threads can also be suspended and subsequently resumed from the same point. These mechanisms can be used to limit the period for which a secondary thread is valid, or to suspend the primary thread while a contingency is being dealt with. A Step represents the finest grained level of control logic partitioning that can be status tracked during execution. Step Confirm Body Trigger Wait for Trigger Check Pre-Condition Initiate Body Body Executes Body Completes Wait to Confirm Check Post-Condition Watch Asynchronous Execution (Thread Blocked, Waiting) Figure 3: Step Phase Wait Condition: Wait For Date-Time, Event or Condition with Timeout Trigger Timeout Precondition Failure Body Initiation Failure Watch Condition Occur. Body Failure Confirmation Timeout Post-Condition Failure Each Step is broken into three clear phases that of triggering, body execution and confirmation shown in Figure 3 Triggering - A Step may have an optional Trigger which controls the synchronisation and checking that conditions are correct to start execution. A Trigger consists of a Wait Condition and a Pre Condition. On activation the Step will await its Trigger Wait Condition. This is followed by the checking of the Pre Condition before progressing on to execute the Body of the Step. Body - Execution of the Step Body involves execution of the intended Step logic e.g. execution of an Application Action or flow control construct such as a loop or branch statement. Confirmation - Once the Body is complete then an optional Confirmation phase of the current step can be executed before moving on to the next Step of the sequence. Various types of Step Body have been defined which implement the different logic and control statements required for operations automation. C. Architecture Recovery Actions Recovery Actions: Associated with Failure Condition Specified at or Step Level (Step Level takes precedence) Figure 4 shows the conceptual architecture that comprises the following major elements: External Definition Graphical Definition Environment TM/TC Definitions Catalogue Serialized Definition. Figure 4: Architecture External Control Initiation & Control Status Execution Manager Execution Execution Engine Parameters Commands Events MCS/EGSE Mission Control or Check-out System Catalogue Initiation / Control Live Overview COS Events Archive Replay Replay Live COS Events Definition Control Overview Display Displays Single Display Definition element provides tools which support the definition, checking and building of operational procedures. Displays element provides the client interactive display components which allow a detailed graphical visualisation of a single procedure and an overview status of all executing procedures. The displays also provide access to the Control Interfaces at the appropriate level. Control Interfaces element provides the networked control interfaces allowing procedures to be executed on request and controlled by the interactive client displays as well as by a non interactive external control source such as a schedule execution system. Execution element provides a lightweight, portable and scalable procedure execution server component. Archive element is responsible for the archive and retrieval of procedure change of state information. 3
D. Definition Environment A powerful graphical editor is provided with which supports the user during the definition of a procedure. Figure 5: Editor The editor supports a rich set of features such as drag and drop step insertion, multi procedure edit, copy, paste, search and mutli action undo/redo. Drop down lists provide easy access to all the parameters, commands and events which can be referenced in a procedure. During an edit session a debugger execution server can be invoked which allows a procedure to be tested offline in the definition environment. The debugger allows the setting of TM parameter values and the specification of the result of actions such as the success or failure of commands. These values can also be saved allowing a re-use in later debugging sessions. E. Execution Environment The Execution Environment supports the execution of s in the context of a particular subsystem of the Space System Model (SSM) referred to as a Domain. The association of s to Domains and Sub-Domains allows the logical partitioning of a particular automated application into a subsystem hierarchy. The concept of Domains also supports the physical partitioning of execution on different server nodes of a computer network. The architecture supports the distributed execution of s by allowing each server node in a network to be configured to host a different sub set of the Domains and Sub-Domains defined in a SSM. The Execution Environment allows the execution of s to be manually controlled by the user. Two levels of manual control are available. With Single Step control the operator must authorize the execution of each step. With Command/ control the operator must authorize the execution of each command or subprocedure step. supports control of execution at the level of the Server Node, Domain and. The networked architecture allows the connection of multiple client display nodes to the server nodes. The client display nodes host the Overview Displays (POD) and the Single Execution Display SPED. The POD shows an overview of Node, Domain and High level information. The graphical SPED can be launched from the POD. It shows the detailed execution status of an executing procedure showing its threads and steps, and automatically Figure 6: Overview Display tracks the current execution point. The 4
SPED uses the same graphical representation as the Editor. F. Archive The Execution Server executes procedures and forwards Change of State events to the Archive. The Archive writes the events to files in the archive area. The Archive provides an API which allows the archived data to be retrieved. The archived data is organized in a way Historical POD Server I Retrieved Events Archived Events Replay POD Archive Server Replayed Events Figure 7: Archive Server II Archived Events Retrieved Events Final State SPED Replayed Events which optimizes the retrieval of the data. The Archive can archive data produced by multiple Execution Servers. It also allows multiple applications to access the archived data. Figure 7 illustrates an example of an deployment in which there are 2 (online) servers and a single Archive Server that has responsibility for archiving the Events from both servers. The Historical POD displays the status at a specified time in the past. The Replay POD dynamically displays the status of s which executed over a configurable time frame in the past. The Final State SPED displays a detailed graphical view of the Final State of a which executed in the past. The Replay SPED dynamically displays the detailed graphical status of which executed over a configurable time frame in the past. A key feature of the design is that archive clients can access the archived data without affecting the Execution Servers indeed the servers need not be present when the data is being retrieved. III. Replay SPED Tailoring & Deployment of for MSG G. MSG Import EUMETSAT had defined a large number of complex s using the UNiT automation system. During nominal operations, the procedures are scheduled using the Mission Planning tool and control all routine operations. This covers operations such as imaging, calibrations, rangings, manoeuvres, eclipses and such like. EUMETSAT had invested significant time & effort in the definition and validation of these s. It was therefore essential that the replacement of UNiT by did not mean that these s had to be redefined. For this reason a Conversion Tool was created to convert the UNiT procedures to the equivalent. This tool is shown in Figure 8. The tool allows the user to select a or group of procedures for conversion. Figure 8: MSG Converter The goal in the production of the converter was to perform a 100% conversion for all procedures. To date the EUMETSAT procedures have been converted as part of System Testing within EUMETSAT. A small number of OL expressions could not be directly converted. This was either due to the 5
OL type definitions/checking being stricter than the original UNiT or due to the detection of inconsistent Constructs which were not previously detected by UNiT OL checking. Overall this dry run of the conversion process was a success with minimal user intervention required The next stage of operational deployment of will be: Familiarisation of the EUMETSAT operations teams with the tools; Formal conversion of the latest procedure set by the EUMETSAT satellite operations team; Rigorous re-validation of the procedures. The revalidation will aim to provide sufficient confidence for EUMETSAT in the functional behaviour of the procedure constructs including OL expressions and the performance of the procedures - as many of the procedures are large, complex and/or with time critical actions. This involves execution of the procedures in as realistic an operational environment as possible, using a satellite simulator. The activity is scheduled to take place in the second half of 2006. Once procedure validation is successful and training complete the software and procedures will be deployed operationally. H. Tailoring In the design of it was foreseen that there were a number of areas where has to be tailored to allow it to be fully integrated into a Space Control System. This tailoring was performed for the MSG system as described below. Definition Space System Model Access Online Node In the definition environment it is necessary to access the Space System Model to allow the Execution s to be populated with Telecommands and TM references. For example when the user created a Execute OL Command step within the he is presented Core with a list of available Telecommands when he /ICOL ICOL Execution Environment selects the required Telecommand, the Telecommand details such as default arguments are copied from the Data Data Action Update Update Injection Space System Model into the. ICOL Data Server API ICOL Action Server API /ICOL defines an open interface through which MSG MSG MSG Data Server Action Server Data/Action Server the information is provided. For MSG therefore it was Parameter Event Command Parameter Event Update Notification Injection Assertion Injection only necessary to provide an implementation of this interface which retrieved the required information MSG Control System from the actual MSG databases. Figure 9: MSG Action/Data Server Online Space System Model Access In the online environment it is necessary to interact with the SSM to inject Actions. Actions are defined in the SSM they are functions which can be performed on objects such as send Telecommand. It is also necessary to interact with the SSM to obtain the (data) values of objects defined in the SSM, such as the engineering value of the parameter. ICOL requires the values SSM objects as these may be operands in OL expressions to be executed. requires these values for online displays. Once again for the MSG/ system it was not necessary to modify the core /ICOL system. It was only necessary to implement the thin Action/Data server shown in Figure 9 which uses the existing MSG CF API to map action/data requests to the MSG CF equivalent. New Step Types In the design phase it was foreseen that when is to be deployed in a new Space Control system there will often be a requirement for Mission specific step types. It was therefore a goal of the design that the introduction of new step types should be as straightforward as possible. This was achieved by encapsulating the step specific information for each step type in a single class. This class must be derived from either the standard step base class or a sub-class of this class. This single class not only provides the specific dynamic behavior required for the step type but it also holds the meta-data used by the Editor to provide a properties pane which allows instances of the step type to be created and modified. For MSG two new step types were defined to provide MSG specific behavior for command and command group injection. The production of these two new step types was straightforward and confirmed that the introduction of new step types into is indeed straightforward. I. Integration with Schedule Execution 6
Figure 10:Schedule Execution Display The existing MSG Space Control System had an advanced Schedule Execution system which is responsible for the initiation of s at the appropriate time. In nominal operations all procedures are initiated by Schedule Execution or as sub-procedures of these procedures. The Schedule Execution display shown in Figure 10 provides the user with a status indication for procedures it has initiated. In the normal mode of execution the Scheduling Execution system will initiate approximately 100 procedures in a 24 hour period. These procedures invoke sub-procedures resulting in a total of 2000 procedures executed in a 24 hour period. To integrate with this component it was necessary to modify the Schedule Execution procedure invocation component to use the socket based high level interface to start procedures and monitor their execution. No changes were required to the high level interface it already provided the actions required by the MSG schedule execution system J. Physical Architecture In system testing two configurations have been adopted:- MSG Ground Station Facilities BRGS PGS MSG Central Facility Schedule Execution M&C s Figure 11: Single Server MSG Single-Server With this configuration a single server controls 7 domains (spacecraft or ground systems). This configuration is shown in Figure 11. Multi-Server With this configuration there is an server for each of the 7 domains. Each server can only Monitor & Control the associated domain however a procedure executing in one domain can invoke sub-procedures in another domain. The availability of this configuration means that if the Satellite Fleet expands the automation system can be expanded and distributed across multiple hardware 7
MSG Ground Station Facilities BRGS PGS MSG Central Facility Schedule Execution platforms without any software changes. This configuration is shown in Figure 12. The availability of these two alternative configurations provides an excellent illustration of the flexibility of the architecture. For simple systems can be deployed as a single server. For larger multi-satellite systems can be deployed on several different servers. The number of servers can be increased as the requirements on the control system increase. This change in the deployment is performed as a simple configuration change. No software changes are required. s s s s M&C Figure 12: Multi Server MSG IV. Conclusion The success of the development so far at EUMETSAT has demonstrated the validity of the following key concepts: Tailorable can be tailored to allow it to be integrated into complex Space Control Systems. Extensible can be extended to provide Mission Specific features. EUMETSAT had a number of mission specific requirements. It was possible to add Mission Specific modules to satisfy these requirements without affecting the core architecture. Scalable As discussed above the distributed architecture means that as a Space Control System increases in size can be distributed across multiple platforms to give the required performance. Client Server the separation of the client & server reduces the impact of client operations on the server. References Reports, Theses, and Individual Papers 1 s language for users in test and operations (PLUTO), ECSS - EUROPEAN COOPERATION FOR SPACE STANDARDIZATION, ESA- ESTEC Requirements & Standards Division, ECSS-E-70-32, Draft. 24, Nov. 2005. EUMETSAT The European Organisation for the Exploitation of Meteorological Satellites, is an intergovernmental organisation that establishes and maintains operational meteorological satellites for 19 European States (Austria, Belgium, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, the Netherlands, Norway, Portugal, Slovakia, Spain, Sweden, Switzerland, Turkey and the United Kingdom). EUMETSAT has signed 11 Cooperating State Agreements. Those with Bulgaria, Croatia, Hungary, Latvia, Lithuania, Poland, Romania, Slovenia and the Czech Republic have entered into force whereas the Agreements with Serbia and Montenegro and Iceland are to be ratified in the near future. EUMETSAT is currently operating Meteosat-6, -7 and -8 over Europe and Africa, and Meteosat-5 over the Indian Ocean. The data, product and services from EUMETSAT s satellites make a significant contribution to weather forecasting and to the monitoring of the global climate. 8