Monitor VisionWorks Landslide System

Transcription

1 Monitor VisionWorks Landslide System

2 Contents Purpose... 3 Application (GUI) System Status... 3 Tree Elements... 3 System View... 4 Test Server View... 4 Test Session View... 5 User ID View... 6 Real-Time log... 6 TS Admin Status... 7 Test Server Status Details... 8 Customize SNMP Traps Using TEX Web Client Additional Information RESTful API Related RESTful API Documentation Nagios Monitoring Overview Access the nagios3 on TAS Configure Nagios Agent:

3 Purpose This document provides administrators of VisionWorks Landslide with fundamental information on monitoring and supporting the system. Though this information is not comprehensive in nature, it can be used to gain an understanding of the monitoring options available, as well as the administration support needed to maintain and troubleshoot the system. The following information contains content available directly within the Landslide User Guide. For detailed information, consult the manual. Information covered within this document includes admin usage of: Application (GUI) System Status Real-time log TS Admin Status Customize SNMP Traps View running tests and server status using TEX Web Client RESTful API to display running tests, test server status, system logs, etc. Nagios Monitoring Application (GUI) System Status The application GUI provides a comprehensive System Status Window with an overall view of: System State Test Operations User Activity Test Server Status information is arranged in an expandable tree hierarchy with color-coded status indicators. The status is constantly updated while the window is open, giving you real-time status without the need to manually refresh the display. System summary information, displayed above the tree view, shows the number of active test sessions, test servers, and SUTs. If POLQA or PEVQ is installed, Remaining POLQA/PEVQ Ecs will be displayed. The tree hierarchy is determined by which of the four available views is selected: System (default), test server, test session, or user ID. The item selected to view forms the root nodes of the tree. A keyhole icon next to a node in the tree indicates that the node is involved in an activity and can be expanded to display further information. Tree Elements The tree nodes display the same information for each element type regardless of the view selected: The System element includes both the TAS Memory In-Use (just the TAS process), TAS Hard Drive Space (just the main partition), Running Tests #, Active Test Server #, Active SUTs #, Remaining ECs (POLQA, PEVQ and DSP) 3

4 The test server element includes the server's host name, the server status and the number of active test sessions running on that server (if any). The test session element includes the session's run ID, name, and state (uninitialized, init, starting, running, stopping, stopped, and cleanup). The user ID element includes the user name and the client platform's IP address. When a user disconnects from a running test session, DISCONNECTED SESSION is displayed instead of an IP address. The SUT element includes the SUT's name and IP address. System View In the System view, the System elements includes: TAS Memory In-Use (just the TAS process), TAS Hard Drive Space (just the main partition) Running Tests # Active Test Server # Active SUTs # Remaining ECs (POLQA, PEVQ and DSP). The memory/disk fields are updated every 30 seconds. They will change color as they reach FULL, similar to the Client Memory Monitor. Test Server View In the Test Server view, all test servers are listed by host name and the version of the software installed on the test server is displayed. In this view the test server status is also indicated with a color code: No color ready and idle (no active test sessions) 4

5 Green ready and active (at least one test session is running) Red not ready (server is running at capacity or is otherwise not available for testing) Orange needs upgrade (test server software is not compatible with the TAS software) Black the test Server Configuration is invalid due to a problem with the assigned IP addresses When a test server is green, the node can be expanded to display the test sessions. The test session nodes can also be expanded to display the session owner and the SUTs that are used in the test session. Test Session View In the Test Session view, all active test sessions are displayed in session ID order. A test session node can be expanded to display the test servers used by the session, the session's owner, and the SUTs that are used in the session. 5

6 User ID View In the User ID view, the user names of all of the users currently logged in to the test system and all those with disconnected test sessions are displayed in alphabetical order. A user node can be expanded to display the client platform's IP address, Client = <client type can be GUI,Tcl, Rest or Cli)>, User = <user name on Client System>, timezone =<Time zone on Client System>, and os = <Operating System on Client System> when the user is logged in to the system. If the user is running one or more test sessions, the client node can be expanded to display the test sessions. Real-Time log A real-time log is available to assist with system troubleshooting and status. Operational messages generated by the TAS and the test servers are reported in the test system's Real-Time Logs. Errors that can affect system operation, and user actions such as changing the definition of a SUT or test server, executing tests, and importing Test Suites are recorded in the logs. The test system maintains three types of logs, and access to these logs is governed by the user's system permissions. The Operations log can be viewed by everyone, and contains system-level messages related to test operations. The System Administration log can be viewed by administrators, and contains changes made to items recorded in the system database. The System log can be viewed by the Super User and System Administrators and contains system errors and messages related to the test system's status. The log messages include information that identifies the message source, the time of generation, the message type, the log type, and the event type. The log messages include information that identifies the message source, the time of generation, the message type, the log type, and the event type. The log messages include information that identifies the message source, the time of generation, the message type, the log type, and the event type. With the Real-Time Logs window, you can: 6

7 View log messages as they are generated Search log messages as they are generated To open the Real-Time Logs window: or or Select View > Real-Time Logs from the Main menu. Click the Real-Time Logs button on the minimized docked panel. Click the Real-Time Logs icon in the Main toolbar. Note: If an error or a warning occurs when the real-time logs are minimized, the Real-Time Logs panel provide a highlighted indication. Click the highlighted indication or the real-time logs toolbar button to open/restore the Real-Time Logs window. TS Admin Status Manage your Test Servers by determining the TS Admin Status. The TAS controls the test system and retains the test definitions, but it's the test servers that perform the tests. The number of test servers that the TAS will recognize is governed by your test system's licensed capacity. The topics in this section will show you how to control the test server's network configuration, the addresses that it makes available for IPv4 and IPv6 test operations, and the software that operates it. To use a Test Server (TS), identify the TS to the TAS by its management port address. The TAS will then be able to direct the test control commands, control the test server's configuration, monitor its status, and upgrade the test server's software. The Test Server Administration window is used to identify the test server to the TAS, view general information, to provide a dynamic view of the test server's Requested License, Ethernet addresses, status, Wi-FI, OTA, UE and installed Certificate files, and to control the test server. To configure a VPN connection between a TAS and Test Server, follow the steps outlined in topic Configure VPN between TAS and Test Server. 7

8 Test Server Status Details The test servers keep the TAS informed of their operational state. Before performing any action that can affect the operation of the test server, look up the test server... or use the System Status window and check its status: Ready 0/Z The test server is currently idle and can accept test or maintenance commands. Where Z is the total number of processes. NOTE: As or release 16, colors have been added to status. Green = #test sessions = 1, Yellow = #test sessions = 2, Orange = #test sessions > 2 Running [Nr] X/Zr The test server is currently running one or more tests, where N is the number of test sessions, X is the number of running processes, Z is the total number of processes. An r after the number of test sessions ([Nr]) indicates Port Reservation in use and an r after the number of running processes (Zr) indicates Process Reservation in use. Maintenance activities could disturb the active operations. The TS status also indicates the mode currently running: Status Indicator What it means... 8

9 RUNNING [2] 1/3 RUNNING [3r] 1/3 RUNNING [3r] 2/3r RUNNING [2] 1/1 RUNNING [3r] 1/1 RUNNING [1r] 1/1r 2 test sessions running in unreserved mode on a TS licensed with Performance Accelerator or Extreme License. 3 test sessions running in reserved ports mode on a TS licensed with Performance Accelerator or Extreme License. 3 test sessions running on reserved Ports + Processes mode, using 2 processes on a TS licensed with Performance Accelerator or Extreme License. 2 test sessions running in unreserved mode on standard TS 3 test sessions running in reserved ports mode on a standard TS 1 test session running in Reserved Ports + Reserved Processes mode on a standard TS TROUBLESHOOT: If the status is Running [0], the test server is running a test but the TAS is not controlling the test. In this case you should recycle the test server to abort the test. Not Ready The test server is not responsive and cannot accept any commands. This status is normal during an upgrade or when the platform is rebooted after a configuration change. When the status is not ready, it could be due to the following states: NOT_READY (NO COMM) The TS isn't communicating with the TAS NOT_READY (NO COMM) The TS isn't communicating with the TAS (TAS-to-TS COMM not confirmed in 30+ seconds is displayed when active check between TAS and Test Server fails to verify connection is available) NOT_READY (HEARTBEAT) The TAS will set NOT_READY HEARTBEAT if no response is received within 30 seconds when active check between TAS and Test Server is checked. Note: If tests are running, they will not be aborted, and the TS will stay RUNNING. NOT_READY (NEEDS RECYCLE) TAS is refusing to accept the status message NOT_READY (PARTIAL TS PROCESSES) User attempted to start a test before all TS processes were READY. For example, TS processes 1 and 2 were registered but 3 and 4 were not before a user attempted to start a test. TROUBLESHOOT: The test server should be Ready within a minute after the platform has booted. Recycle the test server to restart the software if it has not reached the Ready state in a reasonable amount of time. 9

10 If the TS displays a NOT_READY (NO COMM) state, check why your TS is not communicating with the TAS. For example, is the TS disabled, off, in the process of rebooting, recycling, not connected to the network, is the network down, etc. If the TS displays a NOT_READY (NEEDS RECYCLE) state, recycle your TS, as it states, if that does not fix the state, you may need to re-configure your TS. NOTE: A TS could go into a NOT_READY (NO COMM) sub-state from the RUNNING state, if the TS stops sending messages to the TAS for any reason, e.g., TS locked up and recycled slowly dumping core. NEEDS_LICENSE Current License Assigned to the TS is not valid. Select a proper license from the "License selection available" or contact support for assistance. NEEDS_UPGRADE The test server is running, but its software is not compatible with the TAS software. The test server must be upgraded before it can accept test commands. OBSOLETE_HARDWARE The test server hardware is obsolete. NOT_SUPPORTED The test server hardware is not supported. Invalid (see Real-Time Logs) The TAS is rejecting the registration messages sent by the test server due to invalid IP addresses or masks used by the test server. You can search the Real- Time Logs for messages with the text "Invalid registration message received." The cause for the rejection is explained in the log message. Additional NOTES: If a Test Server reports that an Ethernet interface is down, on the Test Server Administration window, the Ethernet interface name will show up in bold-red with a small down-arrow on the left side. If you try to run a test using an Ethernet interface that is down, you'll get an error indicating that the Ethernet interface is down. Configuring tests will not be affected by down status. Running Tests counter now ignores COMPLETED tests. Customize SNMP Traps Enable SNMP Traps for MGMT status reporting from Server Settings SNMP. 10

11 To setup, or modify perform the following steps: 1. Select Admin > Server Settings and the Server Settings window opens. 2. Select to Enable SNMP Traps to Address - Enter the address of the network server that is responsible for processing SNMP Traps. Enables rest of the fields. a. Retransmission Period (min) - Set the period for single retransmission traps in minutes. Traps do not retransmit continuously, just once. (Default: 1) b. Enable License Server Status Trap - Enables Trap sent when there is a change in License Server Licensed State. This trap will be sent only if using license server license. c. Enable TAS Started Trap - Enables Trap sent when TAS fully successfully starts. / TAS Starting (1) sent when starting TAS. / TAS Starting (0) sent when stopping TAS. d. Enable TAS Hard Drive Full Trap - Enables System disk utilization traps. Warning and Error level traps. e. Enables Test Server READY/NOT_READY Trap - Enables Test Server READY/NOT_READY traps. NOT_READY traps will not be sent until the TS is first in a READY state. f. Version SNMPv2 SNMPv3 - Select version SNMPv2 or SNMPv3. g. Community - Available only when version = SNMPv2. Default = public h. Available only when version = SNMPv3: Context Name i. Context Engine Id ii. User Name iii. Authentication Protocol iv. Authentication Password v. Privacy Protocol vi. Privacy Password 3. Get MIBs - Opens browser to URLs required to configure NMS. 11

12 Using TEX Web Client View running tests and server status using TEX Web Client (beta). The Web Client has 3 main functions: Run Test View Running Tests View Test Servers Perform the following steps for usage: 1. Login Using: URL comes in two options: TASIP/tex/ -OR- TASIP/tex/ NOTE: HTTPS is supported for TEX. We recommend installing a proper SSL Certificate prior to use. For additional details on installing an SSL Certificate on the TAS, see Signed SSL Server Certificate in the TAS Manager Menu. It is especially critical if using IPv6 HTTPS to install a proper SSl Certificate. 2. Once logged in, the main tool bar has access to the three main functions. The IP address of the TAS and the user name are underlined below. 3. Select Layout: Standard, Horizontal or Vertical. Horizontal Tab: 12

13 4. The TABs can be dragged and dropped to change the order. Vertical Tab: 5. From Views Ts, get the list of test servers. When one is selected, the "TS-Admin" type information is available and you have the option to Recycle the TS (if you have authority). 13

14 6. View Running Tests to see all the running and completed (but still live) test sessions with some filtering ability: 7. Use filter to see just your own "My Tests" vs "All Tests" and/or see "Running" vs "Completed" vs "All" tests. 8. Select a test to see additional information on the right side and to have the ability to Delete completed tests you own and Reconnect to any test. 9. The Run Test App relies on Test Session Favorites to populate the screen with information. When you run/monitor a test from the TEX Client, there are up to 6 sections. There is always a section at the top for displaying Overall Test State or Step, Overall Pass/Fail Criteria State, command buttons (Start/Stop/Continue/Abort/Delete), and Test Result File Links. If Favorite Measurements are defined, there is a Favorites Report table section. If any Charts/Graphs are defined, up to 5 sections are displayed, one for each of them. For starting a test, if you defined Favorite Parameters from the Cross-Reference, supported parameters can be overridden before you start the test. 14

15 10. After you login go to Admin->View TSs to see TS page, then go to Admin->View Tests to see the test session listing, choose to view "ALL" users and "ALL" states. 11. Then if you click on the ID=2 Test, The right side so extra info about the test. In this example, when test is complete it will list the result files and at the top there are links to the PF Criteria and Measurements RESTful API URLs. And you can "Reconnect" to it, it will load the test in the Test Runner app. Once you do that you can click on the "Results Files " and "Details " buttons to see those pop ups: 15

16 Additional Information The TAS only supports 48 simultaneous logins (i.e. connections). The combination of Tcl API + GUI Clients + REST (Web) Clients must never result in more than 48 client connections to the TAS in the same instant. The TEX Web client uses the RESTful API with non-persistent HTTP, so these connections are temporary, asynchronous and quickly closed, but they still count towards the 48 logins while they are happening. You might get away with more than 48 Web Clients, but at some point, if requests overlap, you will get errors back from the TAS that could affect all Client types. This is accessed via an API and you must DELETE test sessions when you are done with them, or else put your TAS at risk for running out of memory. When you are done reviewing the results of a completed test, click the DELETE button. If you accidentally left one undeleted, you can go to the View Running Tests to delete any tests you completed. Charts/Measurements are queried every 5 seconds, and chart data points only updated if the interval changes or the value changes. Chart history is not loaded when you reconnect to a test, you will only get new data points. Test Status is queried every 2 seconds. 16

17 Pass/Fail Criteria Details are only queried once when you open the dialog, if you leave the dialog open, nothing will change. The only things refreshed automatically are the Test Status/Charts/Favorite Measurements. You cannot command a test session beyond Start/Stop/Abort and Continue (Automation Control Wait step). TEX Web Client does not provide ability to interact with live TC sequencer, DMFs, ODCs, or updates. For those you can use RESTful API directly or else the standard GUI/Tcl ways. RESTful API This RESTful API is for use with your Landslide Performance Test System. This is a licensed product. Landslide provides direct, but limited, access to the TAS organized around REST. This makes automation functionality readily available to a wide variety of clients via HTTP, allowing the use of existing tools, available for many languages and environments, to create Spirent Landslide Automation clients. Automation clients no longer require a Spirent Landslide installation on the client system. The HTTP request methods GET, POST, and DELETE are used with a URI identifying a resource or function to provide access to all Spirent Landslide RESTful functionality. The RESTful functionality only covers the major operations to manage test servers and to start and command test sessions. The creation of test sessions from RESTful is not supported. The modification of test session is supported but limited. NOTE: Each URL begins with: the tas-ip-address specifies the Landslide Manager server to connect to, 8080 identifies the port that the API is served from, and the api portion identifies the service, and following that are the resources to access. Additional NOTE: For HTTPS the port must be and you must accept the Self-Signed Certificate, via a browser exception or your automation's truststore, and optionally fully install our self-signed SSL Certificate. The Certificate will change with each TAS Release/Upgrade. Related RESTful API Documentation Additional documentation items that are related to this manual are listed below. Landslide REST API Reference This document contains reference information about Landslide Restful API system. RESTFul API Functions - RESTful API examples. RESTful API on Postman - Real Examples using Postman. Additional example usages of the Landslide RESTful API are provided on the Landslide Home page and others can be provided by Support. 17

18 Nagios Monitoring Overview Nagios3 is provided on Landslide as of version Nagios3 will work on Ubuntu onwards. Nagios manager is installed on TAS & Nagios agent is installed on TS. Nagios will also install SNMP. Access the nagios3 on TAS The following provides the URL and login credentials to access Nagios. URL: username: nagiosadmin password: Nagios Configure Nagios Agent: Configure Nagios Agent: (On Test Server) Execute: ipcfg (to configure with Nagios Manager - No need to reboot after executing ipcfg) 18