System Monitoring. System Level Monitoring CHAPTER

Transcription

1 CHAPTER 5 System Monitoring The CDSM provides tools that can be used for system monitoring and system diagnostics. The topics covered in this chapter include: System Level Monitoring, page 5-1 Monitoring Content Objects, page 5-4 Monitoring Stream Objects, page 5-14 Array Level Monitoring, page 5-25 Server Level Monitoring, page 5-26 Recommended Monitoring Schedule, page 5-36 Note If Virtual Video Infrastructure (VVI) with split-domain management is enabled, the CDSM pages associated with the Vaults and Caching Nodes display only on the VVI Manager (VVIM), and the CDSM pages associated with the Streamers display only on the Stream Manager. For more information, see the Virtual Video Infrastructure section on page D-5. System Level Monitoring The System Level Monitoring pages provide an overall view of the health and activity of the CDS. The System Level links are: System Health System Snapshot To view the System Level Monitoring pages, click Monitor from any page in the CDSM, and then click System Health or System Snapshot, as appropriate. OL Cisco TV CDS 2.1 ISA Software Configuration Guide 5-1

2 System Level Monitoring Chapter 5 System Monitoring System Health menu takes you to the CDSM page that has more information. For example, in Figure 5-1, clicking System health problems reported Figure 5-1 CDSM Banner System Health Alarm To view the System Health page, choose Monitor > System Health Figure 5-2 System Health Page Green All components are operating. Yellow Some components are not operational. Red All components have failed. You can view the details of a monitored area of a server by clicking the box in the appropriate column. When you click the Network section on page 5-30 for more information. When you click the Disk section on page 5-27 for more information. When you click the Services 5-2 Cisco TV CDS 2.1 ISA Software Configuration Guide OL

3 System Snapshot Figure 5-3 System Snapshot Page Table 5-1 Field System Snapshot Page Description 5-3

4 Monitoring Content Objects Chapter 5 System Monitoring System Snapshot Page (continued) Active Stream Bandwidth Active Fill Bandwidth Total Content Active Ingests Completed Ingests Prov. (push) Ingests Failed Ingests Temp Out of Service Total Disk Disk Used Disk Available Total bandwidth, in megabits, used for active streams. Total bandwidth, in megabits, used for caching content among Vaults and Streamers. Total number of content objects currently stored, ingested, provisioned for ingest, and failed ingest on the CDS. Total number of content objects currently being ingested on the CDS. Total number of content objects currently stored on the CDS. Total number of content objects that have been requested for ingestion, but have not yet begun active ingestion. Total number of content objects that failed to complete the ingest process. Total number of content objects that are in a Temp Out of Service state. The backoffice may put a content object into this state for a certain amount of time. Total disk space, in bytes, on the CDS. Total used disk space, in bytes, on the CDS. Total available disk space, in bytes, on the CDS. Monitoring Content Objects Table 5-2 Monitored Ingest States Package Expiration Publish Failures Lists Package Expiration information, including expiration dates and all associated metadata. Allows for package expiration adjustments. Lists the packages that were not able to be published to the backoffice and provides a mechanism to republish the package. details: Content Name, File Size, Rate, Create Time, and Last Modified. For the other completed ingests fields, see the same content asset on the VVIM. 5-4 Cisco TV CDS 2.1 ISA Software Configuration Guide OL

5 Ingests Step 1 Monitor > Server Level Step 2 Search Ingests Search Back

6 Figure 5-4 Completed Ingests Page 5-6

7 Content Object Details The IP address of the Vault s ingest interface used to download the content. File Size The file size, in bytes, of this content object. Rate The rate of ingest in bits per second ( = 3.75 Mbps). Create Time The time and date this content object was created. Last Modified The time and date this content object was last modified. Op State The operational state of this content object. The possible operational states are: Created Content is loading. In Service Content is available for streaming. Out of Service Content is not available for streaming. Admin State The administrative state of this content object. The possible administrative states are: Unprovisioned Content is loading. In Service Content is available for streaming. Out of Service Content is not available for streaming. Push Provision The type of FTP provisioned. The provision types are: FTP pull FTP push Content Copies These fields display the following information about the copies of the content: Speed/Direction The trick-mode speed and direction (fast forward or rewind). Server ID The server ID where the copy is stored. Status The status of the stored content. Trick Speeds [1-8] Trick speeds 1 through 8 show the trick-mode speeds for this content object. Full Content ID The full identification of this content object. Ior 1 The Interoperable Object Reference (IOR) for this content object. URL 1 The Uniform Resource Locator (URL) address of the content has the following: Protocol used (for example, FTP) Username and password (for example, videolan:mpeg4ftp) IP address of the content provider s server (for example, ) Directory where the content is stored on the provider s server (for example, videolan) Name of the file (for example, long_encore_3.75.mpg) Asset Ior 1 The Asset IOR associated with this content object. 1. Not displayed in Figure

8 List Content Step 1 Monitor > System Level > Completed Ingests Step 2 Only Damaged List All List 5-8

9 Completed Ingests Content List 5-9

10 Package Expirations Note Step 1 Monitor > Package Expiration Available Packages Display 5-10

11 Figure 5-6 Package Expiration Page Step 3 Expiration Update Adjust License

12 Step 4 sign (+) next to the metadata you want to view. To delete a package, choose the package from the drop-down list, click, and then click in the Package Expiration Details section. Publish Failures Note Step 1 Monitor > Publish Failures Step 2 Unpublished Packages Display

13 Chapter 5 System Monitoring Monitoring Content Objects Figure 5-7 Publish Failures Page Table 5-6 Publish Failures Details OL Cisco TV CDS 2.1 ISA Software Configuration Guide

14 Monitoring Stream Objects Chapter 5 System Monitoring Monitoring Stream Objects Stream Monitor Step 1 Step 2 Figure 5-8 Stream Search Results Cisco TV CDS 2.1 ISA Software Configuration Guide OL

15 Figure 5-9 Stream Monitor Page

16 Table 5-7 Stream Details Service Group QAM IP QAM Port Program Number Bandwidth Used Op State Admin State The output transport stream identification on the associated MQAM device. The service group that the stream object is transmitting on. The IP address of the QAM device delivering this stream object. The port the QAM device is using to receive the stream object. This field is only applicable when Streaming Mode is set to ASI. The numerical MPEG program number for this stream object. The transport stream bandwidth, in bytes, required for this stream object. Operational State indicates the state of the object. The possible states are: InService Stream object is functioning. OutOfService Occurs when the entity using the object wants to temporarily stop the object. Created Stream object not yet provisioned. Destroyed Stream object is destroyed. The administrative state of this stream object. The possible administrative states are: Unprovisioned Stream is loading. InService Stream is available for streaming. OutOfService Stream is not available for streaming.

17 Provision Multiple n n n n means, where n means n times rewind Provision Multiple is enabled if this field is yes and disabled if this field is no. The Graph This Stream

18 Figure 5-10 Stream Activity Report

19 Figure 5-11 Stream Failures Page

20 Table 5-8 Stream Failure Details Table 5-9 ID Names in the CDSM GUI and CServer Files CDSM GUI ID Name CServer Files ID Name

21 Chapter 5 System Monitoring Monitoring Stream Objects CDSM GUI ID Name CServer Files ID Name Error Code Description MSA_CMPT_NOT_EXIST 1 MSA_REQ_TIMEOUT 1 MSA_CMPT_OUT_OF_SVC 1 MSA_REQ_NOT_IMPL 1 MSA_RES_INVALID 1 MSA_RES_DUP 1 MSA_CMPT_DUP MSA_REQ_CREATE MSA_UNKNOWN MSA_REAP MSA_VS_LSC_TIMEOUT MSA_SVC_GROUP_MISSING MSA_RES_NO_CAPACITY MSA_RES_NO_BANDWIDTH MSA_REQ_FAIL MSA_RES_UNAVAIL MSA_FLOW_RT MSA_LSC_SERVER_FAILURE MSA_LSC_NO_MEMORY MSA_LSC_IMPL_LIMIT MSA_LSC_NO_RESOURCES A component does not exist. A client timed out waiting for a response to a request. A component is unavailable. The requested item is not implemented. A resource is invalid. A duplicate resource is being added to the session. It was determined that a component that was being added has the same name as a previously created component. An attempt to create a request failed. An unclassified or undetermined error occurred. A component is destroyed outside of the normal expected flows. The video server timed out waiting for the client to issue a resume or play command after the initial creation of the stream. A request contains a missing service group. A resource is currently out of capacity and cannot satisfy the request. A resource does not have the bandwidth to deliver the stream. A request failed. A response is not available. An entry or exit of a measured or tracked flow, and is recorded in real-time. LSC response; server failed. LSC response; dynamic memory allocation failure. LSC response; implementation limit exceeded. LSC response; no resources. OL Cisco TV CDS 2.1 ISA Software Configuration Guide

22 MSA_LSC_SERVER_ERROR LSC response; server error. MSA_LSC_MPEG_DELIVERY LSC response; unable to deliver MPEG stream. MSA_LSC_ERR A generic DSM-CC error event. MSA_LSC_BAD_REQUEST LSC response; invalid request. MSA_LSC_BAD_STREAM LSC response; invalid stream handle. MSA_LSC_WRONG_STATE LSC response; wrong state. MSA_LSC_UNKNOWN LSC response; unknown error. MSA_LSC_NO_PERMISSION LSC response; client does not have permission for the request. MSA_LSC_BAD_PARAM LSC response; invalid parameter. MSA_LSC_NO_IMPL LSC response; not implemented. MSA_LSC_TRANSIENT LSC response; transient error. MSA_LSC_BAD_SCALE LSC response; incorrect scale value. MSA_LSC_BAD_START LSC response; stream start time does not exist. MSA_LSC_BAD_STOP LSC response; stream stop time does not exist. 1. This event is monitored by the CDS as well as MSA, and is displayed with the prefix AVS_ instead of MSA_. Table 5-11 lists the error codes for internal errors and external errors. Internal errors are errors that occurred in the CDS and specifically describe where the error occurred. External errors are errors that occurred in the network or network components, which includes the ContentStore, StreamService, and so on. The error codes listed in Table 5-11 provide more detail to the MSA_INT_ERR or AVS_INT_ERR error code. INT_ERR Error Codes INGEST_THREADS_NOT_RUNNING INGEST_NIC_DOWN INGEST_DATA_BLOCKAGE BAD_CONTENT NOT_ENOUGH_NIC_BANDWIDTH NOT_ENOUGH_SYSTEM_RESOURCES NOT_ENOUGH_DISK_SPACE_AVAILABLE STREAMER_MAX_SLOTS_LIMIT_EXCEEDED REMOTE_VAULT_DOWN REMOTE_STREAMER_DOWN VAULT_DISK_BAD STREAMER_DISK_BAD CONTENT_LOCATE_FAILED CONTENT_FILL_FAILED Cache server threads are not running. Ingest interface is disabled. Ingest data read is backlogged, causing data socket blockage. Content data is not recoverable. Not enough bandwidth left over on NICs to perform the operation. Not enough system resources left to perform the operation. Not enough disk space available. No stream slot available to allocate the stream. The remote Vault is not responding. The remote Streamer is down. The disk is bad on a Vault. The disk is bad on a Streamer. Cannot locate the content on any Vault. Cannot push content to the Streamer from Vault.

23 NOT_ENOUGH_FILL_BANDWIDTH FILL_LINKS_DOWN STREAMING_LINKS_DOWN VAULT_MIRRORING_SITE_DOWN SET_CONTENT_BUNDLE_FAILED SET_DESTINATION_FAILED DESTROY_STREAM_FAILED PLAY_STREAM_FAILED FILLCB_FAILED WAIT_FOR_FTP_DATA_DONE_FAILED GET_CURRENT_NPT_FAILED CAN_NOT_CONNECT_TO_NAME_SERVICE CAN_NOT_CONNECT_TO_NOTIFY_SERVICE CAN_NOT_CREATE_EVENT_CHANNELS NO_CONTENT_EVENT_CHANNEL_FOUND NO_STREAM_EVENT_CHANNEL_FOUND EVENT_CHANNEL_OBJECT_NOT_EXISTS CORBA_CONNECTION_FAILED CORBA_BROKEN_PIPE CORBA_CONTENT_STORE_BIND_FAILED CORBA_STREAM_SERVICE_BIND_FAILED CORBA_SYSTEM_ERROR CORBA_TRANSIENT_ERROR CORBA_TIMEOUT_ERROR CORBA_IOR_NIL CORBA_IOR_NIL_AFTER_NARROW SERVICE_GROUP_NOT_SUPPORTED REMOTE_CONTENT_STORE_FACTORY_DOWN VAULT_HAS_FULL_LOAD FTP_CONNECTION_FAILED FTP_SERVER_BIND_FAILED FTP_PUSH_TIMEOUT FTP_QUIT_RECEIVED_DURING_INGEST NO_LSCP_SET_TOP_CONNECTION LSCP_SERVER_BIND_FAILD LSCP_PROXY_BIND_FAILED Not enough fill bandwidth available. The fill links are down. The stream links are down. The mirroring site of the Vault array is down. Set content bundle descriptor array failed. Set destination of stream failed. Destroy stream failed. Play stream failed. Fill CB failed. Wait for FTP data done failed. Get current NPT for LSCP status failed. Cisco ISA cannot connect to the BMS Naming Server. Cisco ISA cannot connect to the Notify Server. Cisco ISA cannot create event channels. Naming server does not have content event channel. Naming server does not have stream event channel. Event channel object does not exist in Notify Server. CORBA System exception while connecting to other entity. CORBA system exception with broken pipe with other entity. CORBA bind exception while starting Cisco ContentStoreFactory. CORBA bind exception while starting Cisco StreamService. CORBA system exception while connecting to the servant. CORBA system exception with object being transient. CORBA timeout exception. The orb object is nil. The orb object is nil after narrow. Service group is not supported. Remote Cisco ContentStoreFactory is down. Vault is running with full load. Connection to FTP server failed. FTP server can not bind to the port. FTP push timeout (PASV is not served fast enough). FTP server received QUIT request. LSCP server to set-top box connection is down. LSCP server cannot bind to the running port. LSCP proxy cannot bind to the running port.

24 STREAMER_GROUP_MAX_LIMIT_EXCEEDED STREAMER_MAX_LIMIT_EXCEEDED REMOTE_STREAMER_NOT_RESPONDING NOT_ENOUGH_MQAM_BANDWIDTH NO_QAM_FOR_SERVER_ID NOT_ENOUGH_QAM_BANDWIDTH STREAMER_IS_NOT_IN_THE_SERVICE_GROUP STREAMER_HAS_FULL_LOAD STREAMER_IS_NOT_CONNECTED_THAT_QAM INVALID_SERVICE_GROUP CONTENT_CAN_NOT_BE_LOCATED CONTENT_OBJECT_NOT_YET_PROVISIONED STREAM_OBJECT_NOT_YET_PROVISIONED STREAM_OBJECT_IS_OUT_OF_SERVICE STREAM_OBJECT_IS_ALREADY_PROVISIONED CONTENT_OBJECT_IS_ALREADY_PROVISIONED STREAM_SERVANT_OBJECT_NOT_EXIST NO_DESTINATION_QAM_IP_FOUND NO_DESTINATION_QAM_PORT_FOUND FAILED_TO_SET_STREAM_DESTINATION UNABLE_TO_ACCEPT_CONNECTION UNABLE_TO_REGISTER_EVENT_HANDLER CAN_NOT_LOCATE_QAM_IP_FOR_TSID_IN NS_LOG_MONITOR_ERROR AVS_ISA_GENERIC_ERROR CACHE2APP_INITIALIZE_ERROR FILLCB_FAILED DESTROYCB_FAILED WAIT_FOR_FTP_DATA_DONE_FAILED SET_CONTENT_BUNDLE_DESCRIPTOR_FAILED SET_DESTINATION_FAILED SET_ENCRYPTION_KEY_FAILED CREATE_STREAM_FAILED DESTROY_STREAM_FAILED DESTROY_REMOTE_STREAM_FAILED PLAY_STREAM_FAILED The stream count is exceeding the limit for the Streamer group. The stream count is exceeding the limit for a Streamer. The remote Streamer is down. Not enough MQAM bandwidth. Server is not connected to any QAM. Not enough QAM bandwidth. Streamer is not in the service group. Streamer is running with full load. Server is not connected to the QAM. The service group is not returned by session gateway. Content is not found in the related content stores. The content object is not yet provisioned. The stream object is not yet provisioned. The stream object is out of service. The stream object is already in service. The content object is already in service. The remote streamer does not have a servant for stream object. No QAM IP Address is received for the stream destination. No QAM port is received for the stream destination. Some error occurred while setting the stream destination. Cannot accept more TCP connections. Cannot register event handle to serve the TCP connection. Cannot locate the QAM IP addresses associated with TSID IN. The ns_log file is not updating. Some unknown error occurred during execution of the operation. Failed to initialize Cache2App library. FillCB failed during content ingest. Failed while destroying the content bundle descriptor. API wait for FTP data done has returned an error. Failed while setting content bundle descriptor for a stream. Failed while setting destination of the stream. Failed while setting ECM keys for the stream. AVS cache server cannot allocate the stream handle. AVS cache server cannot tear down the stream. AVS cache server cannot tear down stream on remote server. AVS cache server cannot play the stream.

25 Error Code DATABASE_DOWN DATABASE_SYNCHRONIZING_REPLICATION_Q DATA_IS_NOT_IN_SYNC DATABASE_RETURNED_ERROR DATABASE_RECORD_NOT_FOUND DATABASE_CAN_NOT_INSERT_RECORD DATABASE_CAN_NOT_DELETE_RECORD DATABASE_CAN_NOT_UPDATE_RECORD DATABASE_QUERY_SEND_ERROR Description Database is down. Database is synchronizing with replication queue. Database is not in sync with master. The database has returned an error; may be there is no record found. The record is not found in the database. The record cannot be inserted into the database. The record cannot be deleted from the database. The record cannot be updated. Failed to make a query to the database. Table 5-12 lists the Managed Services Architecture (MSA) error codes for the optional Ingest Manager feature. MSA Error Codes for the Optional Ingest Manager Feature Array Level Monitoring The Array Level Monitoring pages provide an overall view of the health and activity of a specified array, as well as provide monitoring and deletion of barker streams. The Array Level links are: Array Snapshot Barker Stream Monitor

26 Array Snapshot Barker Stream Monitor Note Server Level Monitoring

27 Chapter 5 System Monitoring Server Level Monitoring Step 1 Step 2 Disk Monitor OL Cisco TV CDS 2.1 ISA Software Configuration Guide

28 Server Level Monitoring Chapter 5 System Monitoring gigabytes displayed for Total Space, Available Space, and % Used are the sum of all the disks installed on the Vault server. The Linux File System Stats table shows the combined total storage space for the partions of the disk drives, the combined available storage space for the partions of the disk drives, and the percentage of used storage for each combined partition. In Figure 5-13, the hda2 partition has an alarm indicator because the usage has exceeded the user-defined threshold of 40 percent. For information on setting thresholds, see the Setting System Thresholds section on page Table 5-13 describes the information displayed when a disk is selected. Cisco TV CDS 2.1 ISA Software Configuration Guide OL

29 Chapter 5 System Monitoring Server Level Monitoring Field Description S.M.A.R.T OL Cisco TV CDS 2.1 ISA Software Configuration Guide

30 Server Level Monitoring Chapter 5 System Monitoring Disk Monitor Vault Disk Activity Graph NIC Monitor Cisco TV CDS 2.1 ISA Software Configuration Guide OL

31 Chapter 5 System Monitoring Server Level Monitoring Field Description OL Cisco TV CDS 2.1 ISA Software Configuration Guide

32 Server Level Monitoring Chapter 5 System Monitoring Field Description Cache/Fill Bandwidth Cisco TV CDS 2.1 ISA Software Configuration Guide OL

33 Chapter 5 System Monitoring Server Level Monitoring Cache/Fill Bandwidth Page Table 5-15 Cache/Fill Bandwidth Fields OL Cisco TV CDS 2.1 ISA Software Configuration Guide

34 Service Description Services Monitor

35 Chapter 5 System Monitoring Server Level Monitoring Service Server Description OL Cisco TV CDS 2.1 ISA Software Configuration Guide

36 Recommended Monitoring Schedule Chapter 5 System Monitoring Recommended Monitoring Schedule Caution Note Daily Tasks Weekly Tasks Note root Monitoring Tasks for Streamers and Vaults Step 1 dh -h Filesystem Size Used Avail Use% Mounted on /dev/hda1 13G 5.2G 7.0G 43% / /dev/hda6 20G 16G 4.3G 78% /arroyo/log Cisco TV CDS 2.1 ISA Software Configuration Guide OL

37 a. find /arroyo name core* find /home/isa name core* b. c. find /root name *.iso find /arroyo name *.iso su - isa show_calypso_services grep -i Link grep i Link /proc/net/pro_lan_adapters/*.info grep -i Speed grep i Speed /proc/net/pro_lan_adapters/*.info grep -i State grep i State /proc/net/pro_lan_adapters/*.info /home/stats/ifstats Step 5 netstat an grep 9999 Step 6 a. tail f /arroyo/log/protocoltiming.log.{date} grep retransmissions Step 7 grep i warning /arroyo/log/protocoltiming.log.<date> more Step 8

38 Recommended Monitoring Schedule Chapter 5 System Monitoring Monitoring Tasks for Vaults 1. < > grep Capacity Disk: tail -f /arroyo/log/streamevent.log. d. Cisco TV CDS 2.1 ISA Software Configuration Guide OL

39 Chapter 5 System Monitoring Recommended Monitoring Schedule Other Tasks OL Cisco TV CDS 2.1 ISA Software Configuration Guide

40 Recommended Monitoring Schedule Chapter 5 System Monitoring Cisco TV CDS 2.1 ISA Software Configuration Guide OL