Ground Segment Monitoring and Controlling using IMS (Integrated Manager Subsystem)

Transcription

1 SpaceOps 2006 Conference AIAA Name: Dr. Peinado, Tracking N.:55742 Ground Segment Monitoring and Controlling using IMS (Integrated Manager Subsystem) Dr. Osvaldo Peinado GSOC-DLR Oberpfaffenhofen, Münchener Str. 20, 82234, Germany The IMS is an integral subsystem of the German Space Operations Centre (GSOC), which hosts the Columbus Control Centre (Col-CC). It is to be used for many different projects, including the generic multi-mission capabilities offered by GSOC in support of spacecraft monitoring and control. The software architecture of IMS allows flexible and scalable configuration to support different types of projects. The IMS is designed to monitor, control and configure the whole Columbus ground center located in Oberpfaffenhofen, with distributed main sites in Germany, USA, France and Russia, and user operation centers all over Europe. The IMS provides overview monitoring and centralized control for the subsystems of the Col-CC ground segment. The subsystems are connected to the IMS via their Element Managers or SNMP, which aggregate the subsystem monitoring and control data and provide this information in an overview form to the IMS. The IMS offers the required GUI elements to display this data. The IMS central management tasks include monitoring, trouble ticket management in a central location and partial control of the following subsystems: DaSS (TM/TC), MVDS (Video) and MCS (Via EM-Element Managers) SAN, NIS, IGS R&N (infrastructure and network, via L1 servers) VoCS (Voice, via SNMP) WAN (indirectly via L1 or SNMP) LDAP user authentication Export of Unified Views via Web Export (ATV-CC, Col-CC, Redu parameters and USOCs) The IMS also processes Ground Ancillary Data (available via the DaSS) and offers endto-end communications oversight displays (unified views) to: FCT (Flight Control Team) All external sites connected to the IGS network In other words, IMS provides to the operator a total monitoring and control of the Ground Segment and allows the external user to monitor its own subsystem. 1 Copyright 2006 by DLR (Deutsches Zentrum fuer Luft-und Raumfahrt). Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

2 I. Introduction he IMS provides centralized overview monitoring and control for the subsystems of the Col-CC (Columbus T Control center) ground segment, located in Oberpfaffenhofen, close to Munich. The subsystems are connected to the IMS via their element managers using different communication protocols, like SNMP, DaSS (Data Service Subsystem) or IMS internal protocols. The IMS makes available the corresponding GUI s (graphical user interface) to display this information. IMS is a very flexible and dynamic system; it can be easily configured for new approaches, like mission support (Soyus increments or new satellites for multi-mission) or new user-centers or new applications, including audio, video and data. The system can provide monitoring information to operators from other teams like the flight control team or the simulation team, via proxy to other networks. The same or dedicated information can also be provided to all the centers connected to Col-CC via the WEB export (a http interface based on Applets) on the OPS-Support network. The access control, roles and user identification is made by LDAP (Lightweight Directory Access Protocol). Besides the extensive monitoring capabilities, the control of the system is reserved only for the ground control team. II. Description of the system From the Col-CC top level (also used as unified view) the whole network and the distributed centers can be monitored and commanded. The figure 1 below shows the top level of the ground control system. Figure 2: IMS top level 2

3 The graphic in the figure 1 shows the prime and backup centers of Col-CC with the different subsystem, the connected centers and the monitored elements of the system. On the top level all the subsystems indicate its own summarized status. To show this top level status, derived parameters are calculated for every subsystem resuming the status of the whole subsystem. The philosophy for the calculation of such parameters is worst win that means, that if something is wrong in a component of the subsystem, the operator can immediately see the status of the sub system on the top level and navigate through the subsystem till he find where the problem is originated. Depending of the severity of the malfunction, different algorithms are applied to produce a red, orange or yellow color, indicating how important the failure is for the system. The operator has also the help of the SMD (System Message Display) with an associated alarm (that can be triggered depending of the severity of the fault), where the cause of the problem is also described in detail. The SMD also shows SNMP traps generated from the central and the remote nodes in case of a failure in the communication with a center. On the top level, the partner centers are located on the left hand side, the USOC s (User Support and Operation Center) on the top and right hand side and the ESC (Engineering support center) on the bottom. Just pressing the button of the desired center or service will open a new GUI with a higher level of detail and with commanding possibilities. The different subsystems, video (MVDS, the Mpeg2 video distribution system), audio (VoCS, the voice conferencing subsystem), data (DaSS, the data service subsystem), network (IGS, International ground segment, and NIS, network infrastructure subsystem), TM/TC (MCS, the monitoring and control subsystem), the archive and workspace subsystem (SAN, the Storage Area Network) and IMS itself can be monitored navigating through the system. For example the figure 2 shows the Voice system top level of the central node. For each center its own Voice system interface can be also monitored with a detailed GUI. Figure 4: VoCS top level at Col-CC 3

4 The International Partners like Houston, Huntsville, Moscow and Toulouse as well the USOC s and ESC s can also be monitored. The services and also the IGS (International Ground Segment) nodes itself can be monitored in detail, for example, providing the operator with the temperature, humidity, the communication links and the general status of the racks as well as the status of the ports of the routers and switches of the different subsystems allocated on the IGS nodes till the firewalls. The status and the commanding features for the external sites are also given on the subwindows. Figure 5: location of the keysets in a room The figure 2 shows an example of one of the subsystems, in this case it is the voice System (VoCS) located at the central node in Col-CC. In this GUI the operator has the possibility to configure and monitor the voice matrixes, as well as the recorders, the PABX connections and the analog and digital switches. By pressing on each of the colored boxes, a new window will be opened with the feasibility to monitor and to command different parts of the system, like start/reset or stop the recorders, swap the matrixes or start/stop the element manager. The status of the venixes, remote and local keysets is also shown in this display, as well the status of the different switches of the system. One example of the many sub windows of this subsystem (voice); is shown in figure 3. Here the operator has the capability to identify the physical allocation of the different keysets and its status in case of malfunction. The operator can see also on the lower left corner of the monitoring GUI the status and the last trap generated by the voice system. The SNMP traps are also shown on the SMD of the IMS and can trigger an alarm. The communication between the element manager (EM) of VoCS and IMS is done through the SNMP gateway using the SNMP v3 protocol. The IMS permanently polls the status of the voice system, resumed in its own element manager and the voice system automatically generates traps when specials events occur. These traps are received and displayed by the IMS. Another example of the monitoring capabilities of the IMS is shown in figure 4; here the memory usage of the different IMS servers is displayed graphically. The hard-disks and network connections of the different servers can also be monitored. In figure 4 the IMS hardware itself is monitored. Via the network infrastructure subsystem (NIS) or the SAN, the hardware monitoring of other subsystems can be displayed in a similar way. Figure 5 shows another example of monitoring and commanding capabilities of the IMS. In this NIS sub window, the central node and the associated user centers are represented for the ISDN connections only. The ATM and other types of connections have their own displays. On the ISDN overview, the operator can see the status of the connections for each center, the routers and dialer of the corresponding center, and also it is possible to start/stop 4

5 and/or check the lines. In only one GUI the operator can also recognize if the desired center is connected to the prime or backup Col-CC center and start or stop the OPS or OPS-SUP connections independently. The IMS has many different GUI s and many ways to monitor a subsystem, for example if the DaSS data reception from Houston fails, it will be shown on the DaSS, on the MCS, on the NIS and on the SMD simultaneously. The operator can follow the fault through the system and easily identify where the problem really occurs, if is a DaSS problem, or a connection problem or something else and take corrective action or inform the corresponding subsystem engineer for troubleshooting. The IMS arquitecture is based on a distributed system, with many agents in different networks and interfaces with all the subsystems on the Col-CC and its wide area network. The core module of IMS receives, processes and prepares the information to be displayed. IMS run in different modes: Operations, Development and Simulations and can support many different users with different roles simultaneously (access control is provided by LDAP, as for all the Col-CC subsystems). On the development environment new scripts and GUI s can be created, tested and after that, promoted to the operational system. This environment is fundamental, due to the flexibility of the system, new GUI s, monitoring scripts and commands are created or adapted in almost a weekly basis. Figure 6: memory usage of the different IMS server process The integrated manager subsystem has many other tools to support the development or adaptation of the system like the GUI Builder (Used to create dynamic GUI elements), the Application Builder (used to configure server type applications), the Timeline Editor (used to configure the timeline), the Reporting Designer, the Parameter Viewer, the System Editor and the Generic Monitoring and control tools (to be used for debugging, troubleshooting and development). On figure 6 the overview monitoring (OVM) is presented. The overview monitoring display presents a synoptic status of the IMS performance. Each box represents an IMS server process. Redundant processes are grouped by frames. The links between the frames always refer to the active instance of a redundant pair. The whole IMS system is running in hot redundancy, excepting the WEB process. Here both processes are active simultaneously and they 5

6 communicate to each other using an internal IMS protocol. That is because the Web export is based on applets and if an external user lost the connection, using a standard hot redundancy the flow of information to the applet will get lost. Opening the IMS OVM, a vertical task bar is also opened, as in each other GUI. In this case the buttons in this bar provides other functions like: Hardware overview (of the IMS servers, see figure 4), Timeline View, Configuration View, Consoler Users View, Exit Button, Start of Configuration Manager Client, Generic monitoring display and Generic control display. The OVM has also a horizontal bar representing all the subsystems and centers of the network. Clicking on the desired subsystem a new window and a vertical task-bar will be opened. On the vertical pop-up menu many sub windows can be opened in order to get a more detailed monitoring of the subsystem. In each sub window exists the possibility to have an even more detailed monitoring, up to the SNMP level, in order to give the operator the ability to see what happens in every single element of the whole system. The entire information is handled in a tree form, and the operator has the possibility to navigate up to the leaves to see every parameter of the system. Figure 7: ISDN commanding and monitoring The monitoring control engine (MCE), located on the middle of the OVM GUI, is the core process of IMS; it receives data from all other IMS servers and the element managers from all the subsystems. The MCE itself is monitored by the OVM. The GADP (Ground Ancillary Data Provider) is responsible for retrieving data from the DaSS and is also able to send data to the DaSS. It can process together monitoring data retrieved from the MCE and from the DaSS and then provide back to the DaSS. The GADP supports two protocols toward the DaSS: first, the DaSS-S for sending IMS data to the DaSS and second, the DaSS-C for receiving DaSS processing data. 6

7 From the four DaSS services, namely: process data, path telemetry, bitstream and telecommanding, the GADP supports only the processed data format. The data types of the processed data format map quite well to the IMS data types provided in the IMS monitoring tree. The GADP allows sending portions of this data tree, as received by the MCE and possibly post-processed towards the DaSS. Also, processed data packets can be subscribed via the UMI (Unified Measure Identifier) mechanism of the DaSS. This data values are inserted into the GADP data tree and forwarded to the MCE. These data is handled like all other monitoring data within the IMS and allows also to set up correlations between DaSS-received processed parameters and monitoring data acquired via the Col-CC element managers. Figure 8: Overview monitoring of IMS For the development of new procedures or GUI s the DEV (Development server) can be used. The DEV is a proxy application to the main MCE server, which serves as a testing environment for procedures and data processing functions. It receives all data from the MCE, but has no command path toward the server. In this way, there is no risk of accidentally shutting down any operational software via commands send to the DEV. One of the most important modules of IMS is the history data bank (HDB); the archiving server of the IMS. It receives all monitoring data and all messages and stores the monitoring data in flat files and the messages in a MySQL database. It is monitored together with the MySQL database in the lower right corner of the IMS overview monitor in figure 6. Since the MySQL database is a cold standby component it is shown only once. The HDB process itself, on the other hand, is hot standby and both instances are monitored and can be controlled. Both, the writing of monitoring data to the flat files as well the storage of messages in the relational database system require a configuration according to the storage needs. The MySQL database is required for the storage of the daily message data in daily tables and for the access to the historical messages. The HDB process supports temporary downtimes of the MySQL database, during this time the HDB buffers the messages in memory. Therefore, the time frame depends on the available memory of the HDB and the rate at which messages are generated. With normal messages loads of a message every 7

8 second, the HDB can survive for at least 30 minutes in the normal setup. This is the time during which the MySQL should be brought back on-line. To monitor the MySQL, a database connection is established from the MCE to the MySQL server. If the connection can be established, the MySQL DB is declared alive. The result of this test is displayed in the lower right corner of the OVM. Figure 9: ATV-CC Unified View The HDB software processes start up depends of the servers startup order, one of the two will be the primary instance and the other one will be the backup instance. The decision is based on the configuration of the two servers; one is the dedicated prime and the other redundant. If both servers are up and running, the prime instance will share all its data with the redundant instance over a real-time data link. This means that any data received by the prime server will be given to the redundant server as well. In addition to this real-time data link, the two HDB servers use the communication channel also to exchange status messages. These contain the health state and the role (prime or redundant) of each server. This allows reacting to situations where the two servers lose their connection. To ensure that all clients recognize the new server, the two server processes are hosted on a pair of hardware servers, which share a common IP address. From the point of view of the client only a very short temporary disconnection is visible. All the other processes in IMS use a similar hot redundancy concept except for the WE (Web Export). The DQM (Data Quality Monitoring) is a satellite process of the MCE server process. It receives all data that it requires for the quality processing from the central MCE and is able to perform calculations based on loaded data. The resulting data quality can be exported again to the MCE and according to the results, start some alarms or indicate a deviation on the expected values. The rules and algorithms to decide the quality of the data can be dynamically changed and adapted to the needs of the system and type of data. The SNMP Gateway (SNMP-GW) translates between SNMP and the IMS internal communications protocol for those subsystems whose element managers are unable to use the IMS internal protocol. Since this communication is critical the SNMP gateway is configured as a hot standby component. The SNMP is used basically for the 8

9 monitoring and commanding of the Voice system and use the same kind of probes that the L1 server. This last server (are more than 12 actually distributed through the system) is an extra sub window of the IMS and uses PERL probes and SNMP gets and traps to monitor the SAN, the NIS and the IGS nodes. This is also a very critical component of the monitoring and commanding system and also hot redundant. With this component is possible to know the status of all the connections between the nodes and the status of the different components on the remote nodes till the SNMP level, providing highly detailed information of the subsystems. The LOG GW (the Log Gateway) process is an IMS server process responsible for reading the syslog-ng (syslog next generation) log files and making a subset of the log messages available to the IMS message system. This allows extracting important messages from the log and showing them to the IMS operators. The rules for the extraction of the log messages are expected to be refined during operations. The WE (Web Export) is a proxy application to the MCE, which serves as secure remote access point for the Web Export Java Applets which are used to display unified views of the Col-CC via HTTP. The WE use the standard internal IMS protocol to communicate to the MCE. An example of the Unified Views is shown on the figures 1 and 7. The Unified Views (UV) can be exactly like a GUI or a reduced part of them or also a special design. The IMS has the capability to generate UV for every center. The UV provides monitoring, but not commanding to the external users. Each external user can have his own UV, the access rights and user accounts are granted by the LDAP, were all the users credentials and permissions are stored. This LDAP server and its functionality are also monitored by the IMS with a special module dedicated for such purpose. The Apache HTTP is required for delivering the web pages which contain the Java Applets. These applets connect to the WE process to display UV with an overview link status of the Col-CC. The http is controlled via the operating system and is only monitored by the IMS server software. In case of a failure, The IMS switches to the redundant instance of the WE and initiates a maintenance work to find the problem related to the failed http. To check the http availability, the MCE server connects to the apache http port and if this connection can be established, the server is declared alive. This check is performed regularly and the results are displayed on the IMS overview monitor. The Proxy is a process that works in a similar way as the WE, but is used to provide information for internal users in Col-CC, like the flight control team located on the control rooms or the simulation team. To provide the information the proxy process runs on the proxy servers that allow passing the information from one network to another using the IMS internal protocol. Here are also only monitoring capabilities provided. III. Conclusion In this paper the Integrated Manager System (IMS) used in Col-CC for monitoring and control of the entire system was described. An overview of the system and the monitoring possibilities for audio, video and data, with a more detailed example of the voice system was explained. Also a thorough description of the IMS itself was presented, describing the different modules, its functionality, and the communication with the different element managers of all the subsystems of the ground segment of Col-CC and the associated partners and users. It is also appropriate to remark, that the IMS is a living system, that it is continually adapted to the new missions and projects of the Col-CC to provide always a precise, reliable, centralized monitoring and commanding system for the ground control team. 9