RMS Alarms. Fault Manager Server Alarms

Transcription

1 Fault Manager Server Alarms, page 1 Upload Server Alarms, page 5 Upload Server Syslog Messages, page 10 PMG Alarms, page 11 PMG Server Syslog Messages, page 15 PAR System Alarms, page 16 PNR System Alarms, page 16 PNR Syslog Messages, page 17 BAC Alarms, page 17 BAC Syslog Messages, page 19 for AP, page 20 Fault Manager Server Alarms In Cisco RMS, the Centralized Fault Management (FM) Framework feature is used to report the alarms from the following RMS components to the network management systems (NMS) such as, Prime Central: Provisioning and Management Gateway (PMG) Log Upload Server (LUS) Alarm Handler The FM server reports framework-specific RMS component reboot alarm to the NMS indicating that the outstanding alarms from that specific RMS component are pending to be cleared, as the RMS component is rebooted. This table lists the alarm description, conditions, and the severity for FM server reporting RMS component reboot alarm and the watchdog reporting FM Server Down / Up and Watchdog Down / Up alarms. 1

2 Fault Manager Server Alarms Table 1: Fault Manager Server Alarms and Watchdog Server Alarms 1. Alarm Name Watchdog Process is not running Value = Watchdog Server is down Value = 1, Critical Value = 26, SW program abnormally terminated Value = Watchdog Service is not available to report alarms Value = WD-WATCHDOG-DOWN Probable Central node watchdog service abnormally terminated Critical (1) This alarm occurs whenever the watchdog service is stopped or is not running on the Central node. Check the status of god service in central node. [root@rms-aio-central admin1]# service god status The server is not available (or you do not have permissions to access it) Start the god service if it is not running. [root@rms-aio- central admin1]# service god start. 2. Alarm Name FM Server is down 2

3 Fault Manager Server Alarms Value = FM Server is down Value = 1, Critical Value = 26, SW program abnormally terminated Value = FM Server service is not available to report RMS alarms Value = WD-FM-SERVER-DOWN Probable Central node FM-Server application abnormally terminated Value = 6, Clear Critical (1) This alarm occurs whenever the FM-Server application is stopped or is not running on the Central node. Check the status of FM Server in the Central node [root@rms-aio-central admin1]# service god status AlarmHandler: up FMServer: unmonitored PMGServer: up [root@rms-aio-central admin1]# service god status The server is not available (or you do not have permissions to access it) Start the god service if it is not running [root@rms-aio-central admin1]# service god start Start the FM-Server running if it is not running or left unmonitored [root@rms-aio-central admin1]# god start FMServer Sending 'start' command The following watches were affected: FMServer 3. Alarm Name PMG Component is rebooted: Clear all outstanding alarms from this component 3

4 Fault Manager Server Alarms Value: PMG Component is rebooted: Clear all outstanding alarms from this component Value = 24, Software error Value = No Service Impact Value = FM_FRAMEWORK_COMP_REBOOT_ALARM_PMG Probable PMG application got rebooted This alarm is a clear alarm which notifies that the PMG component in RMS got rebooted. Clear (6) This alarm occurs whenever PMG application is started. This alarm does not require any recovery because it is a boot notification to the trap manager. 4. Alarm Name LUS Component is rebooted: Clear all outstanding alarms from this component 4

5 Upload Server Alarms Value: LUS Component is rebooted: Clear all outstanding alarms from this component Value = 24, Software error Value = No Service Impact Value = FM_FRAMEWORK_COMP_REBOOT_ALARM_LUS Probable LUS application got rebooted This alarm is a clear alarm which notifies that the LUS component in RMS got rebooted. Clear (6) This alarm occurs whenever LUS application is started. This alarm does not require any recovery because it is a boot notification to the trap manager. Configuring SNMP s Refer to "Integrating RMS with Prime Central NMS Post OVA Installation" in the RMS Installation Guide for information regarding the use of the configure_fm_server.sh script. This script is used to configure the SNMP manager configuration. This configuration would handle PMG, Upload Server, Fault Manager and Alarm Handler alarms. Upload Server Alarms In Cisco RMS, the Centralized Fault Management (FM) Framework feature is used to report Upload server alarms to the NMS systems, such as Prime Central. This feature supports the SNMP alarm reporting using the CISCO-EPM-NOTIFICATION-MIB. The following tables list the Upload server alarms with their OIDs, probable cause,, clear alarm information, severity, reason for alarm trigger, and recovery information. 5

6 Upload Server Alarms Table 2: Upload Server Alarms 1. Alarm Name Upload Server terminated Value = Upload Server terminated Value = 1, Critical Value = 26, SW program abnormally terminated Value = Upload Server is not available to handle log upload requests Value = FC_LUS_SHUTDOWN Probable Upload Server abnormally terminated. Critical (1) This alarm occurs whenever the Upload server process is stopped or is not running on the Upload node. [root@rms-aio-upload admin1]# service god status UploadServer: unmonitored [root@rms-aio-upload admin1]# service god status The server is not available (or you do not have permissions to access it) Upload node. Start the god service if it is not running. [root@rms-aio-upload admin1]# service god start Start the Upload server if it is not running or is left unmonitored. [root@rms-aio-upload admin1]# god start UploadServer Sending 'start' command The following watches were affected: UploadServer 6

7 Upload Server Alarms 2. Alarm Name Watchdog Server on Upload Node is down Value = Watchdog Server on Upload Node is down Value = 1, Critical Value = 26, SW program abnormally terminated Value = Upload Server will not be made available to provide service Value = FC_LUS_WD_SHUTDOWN Probable Watchdog Server on Upload Server abnormally terminated Critical (1) This alarm occurs whenever the Upload server process is stopped or is not running on the Upload node. Check the status of Upload server in the Upload node. [root@rms-aio-upload admin1]# service god status The server is not available (or you do not have permissions to access it) Upload node. Start the god service if it is not running. [root@rms-aio-upload admin1]# service god start 3. Alarm Name LUS disk utilization will be high soon 7

8 Upload Server Alarms Value = LUS disk utilization will be high soon Value = 8, Informational Value = 16, Threshold crossed Value = ULS performance for handling log upload REQs might be impacted Value = FC_LUS_DISK_UTILIZATION_INFO Probable The disk space of Upload node is getting filled up by the FAP statistics logs. Warning (4) The disk space in Upload server has crossed the information threshold limit configured in the UploadServer.properties file. Remove the old archived statistics or logs from the Upload node. 4. Alarm Name LUS disk utilization will be high soon 8

9 Upload Server Alarms Value = LUS disk utilization going high be warned Value = 4, Warning Value = 19, Resource at or nearing capacity Value = Upload Server performance might be impacted in near future Value = FC_LUS_DISK_UTILIZATION_WARNING Probable The disk space of Upload node is getting filled up by the uploaded logs. Warning (4) The disk space in Upload server has crossed the warning threshold limit (default = 80%) configured in the UploadServer.properties file. Remove the old archived statistics or logs from the Upload node. 5. Alarm Name LUS disk utilization in critical state 9

10 Upload Server Syslog Messages Value = LUS disk utilization in critical state Value = 1, Critical Value = 16, Threshold crossed Value = ULS performance for handling log upload requests is impacted Value =FC_LUS_DISK_UTILIZATION_CRITICAL Probable The disk space of Upload node is getting filled up by the uploaded logs. Critical (1) The disk space in the Upload server has crossed the warning threshold limit (default = 90%) configured in the UploadServer.properties file. Remove the old archived statistics or logs from the Upload node. Upload Server Syslog Messages This table lists some of the Upload Server conditions and the corresponding syslog messages generated as output for the Upload Server alarms. Table 3: Upload Server Syslog Messages Conditions Upload Server process has started. Upload Server process has stopped. Upload Server disk usage exceeds the INFO threshold Syslog Messages UploadServer [trigger] process is running (ProcessRunning) UploadServer has stopped "AlarmEvents Total disk utilization: exceeds INFO threshold" 10

11 PMG Alarms Upload Server disk usage exceeds the WARNING threshold Upload Server disk usage exceeds the CRITICAL threshold "AlarmEvents Total disk utilization: exceeds WARNING threshold" "AlarmEvents Total disk utilization: exceeds CRITICAL threshold" PMG Alarms In Cisco RMS, the Centralized Fault Management (FM) Framework feature is used to report Provisioning and Management Gateway (PMG) alarms to the network management systems (NMS) such as, Prime Central. This feature supports the SNMP alarm reporting using the CISCO-EPM-NOTIFICATION-MIB. This table lists the PMG alarms with their descriptions, conditions, and severity. Table 4: PMG Alarms 1. Alarm Name PMG Server terminated Value = PMG Server terminated Value = 1, Critical Value = 26, SW program abnormally terminated Value = PMG Server (RMS Provisioning Requests Handler) is not available Value = FC_PMG_SERVER_SHUTDOWN Probable PMG Server terminated abnormally. Critical (1) This alarm occurs whenever the Upload server process is stopped or is not running on the Upload node. 11

12 PMG Alarms Check the status of PMG Server in the Central node. admin1]# service god status AlarmHandler: up FMServer: up PMGServer: unmonitored admin1]# god start PMGServer Sending 'start' command The following watches were affected: PMGServer 2. Alarm Name Number of connections to PMG reached the threshold limit Value = No of connections to PMG reached the threshold limit Value = 4, Warning Value = 19, Resource at or nearing capacity Value= Possibility of RMS Provisioning Requests getting dropped Value = FC_PMG_CONNECTION_LIMIT_WARNING Probable Lot of provisioning operations are carried out via DCC-UI /PMG NB API. Warning (4) This alarm is raised when the number of PMG NBI connections crossed 75% of the threshold limit specified in "pmg.max.connections" of the "PMGServer.properties" file. Bring down the north-bound connections by reducing the provisioning operations or increase the connection limit in the "pmg.max.connections" of the "PMGServer.properties" file. 12

13 PMG Alarms 3. Alarm Name Number of connections to PMG exceeded the limit. New connections will be rejected. Value = No of connections to PMG exceeded the limit. New connections will be rejected Value = 2, Major Value = 16, Threshold crossed Value = RMS Provisioning Requests are expected to be dropped Value = FC_PMG_CONNECTION_LIMIT_EXCEEDED Probable Lot of provisioning operations are carried out via DCC-UI /PMG NB API. Major (2) This alarm is raised when the number of PMG NBI connections crosses the threshold limit specified in "pmg.max.connections" of the "PMGServer.properties" file. Bring down the north-bound connections by reducing the provisioning operations or increase the connection limit in the "pmg.max.connections" of the "PMGServer.properties" file. 4. Alarm Name RDU connection lost. PMG cannot connect to the RDU. 13

14 PMG Alarms Value = RDU Connection Lost PMG cannot connect to the RDU Value = 2, Major Value = 30, Application subsystem failure Value = RMS Provisioning Requests Handling Failure Value = FC_PMG_RDU_CONNECTION_LOST_ALARM Probable RDU is not running, is not reachable, or is wrongly configured for PMG. Major (2) This alarm is raised whenever PMG is unable to contact RDU. Check the status of RDU on central node [rms-aio-central] /home/admin1 # /etc/init.d/bpragent status BAC Process Watchdog is running. Process [snmpagent] is running. Process [rdu] is not running. Process [tomcat] is running. Start RDU if it is not running [rms-aio-central] /home/admin1 # /etc/init.d/bpragent start rdu Process [rdu] has been started. If RDU was running already, check and verify the RDU configurations in dcc.properties file femto.bac.rduhost=localhost femto.bac.rduuser=dccapp femto.bac.rdupassword=jsqjwt5ayvu\= 5. Alarm Name PMG unable to make radius calls 14

15 PMG Server Syslog Messages Value = PMG Unable to make radius calls Value = 1, Critical Value = 9, Communications protocol error Value = FAP provisioning failures(whitelist update/deregister/shutdown) Value = FC_PMG_RADIUS_NOCONNECION_ALARM Probable HNB-GW is not reachable or is not configured for Radius calls. Critical (1) This alarm is raised whenever PMG is unable to contact HNB-GW. Check whether the HNB GW Server configuration in the FemtoGateway group of the device. Add the properties if they are missing or configured incorrectly. Check whether the HNB GW Server is reachable and available to RMS. Ensure that the HNB-GW Server is up and running and reachable from RMS. PMG Server Syslog Messages This table lists some of the PMG server conditions and the corresponding syslog messages generated as output for the PMG alarms. 15

16 PAR System Alarms Table 5: PMG Syslog Messages Conditions PMG Server is stopped on request PMG Server is started PMG Server is terminated Syslog Messages (examples) PMGServer [2961]: PMGServer has stopped by request (watchdog may restart it) PMGServer [trigger] process is running (ProcessRunning) PMGServer [trigger] process 3062 exited PAR System Alarms This table lists the Prime Access Registrar (PAR) system alarms that are triggered during the monitoring of the processes. Table 6: PAR System Alarms OID Description PAR Sever stop Server has stopped normally on the host from which this notification is sent Critical (1) PAR Server start Server has started Clear (6) CAR input queue full Indicates that the percentage of usage of the packet input queue has reached its high threshold of 90% Warning (4) CAR input queue not very full Indicates that the percentage usage of the packet input queue has dropped below the low threshold of 60%. Warning (4) CAR license usage Indicates the percentage of license usage. This trap is generated when CAR server reaches license usage thresholds for 80%, 90%, 100%, and 110%. Warning (4) For details about configuring SNMP destinations for receiving PAR Alarms, see the "Configuring the SNMP and SNMP Servers" section in the RMS Installation Guide and use the configuresnmpservingnode.sh script for setting up the trap destination. PNR System Alarms This table lists the Prime Network Registrar (PNR) system alarms that are triggered during the monitoring of the processes. 16

17 PNR Syslog Messages Table 7: PNR System Alarms OID cisconetregserverstop cisconetregserverstart Critical (1) Clear (6) For information about the PNR syslog messages, see PNR Syslog Messages, on page 17. For details about configuring SNMP destinations for receiving PAR Alarms, see the "Configuring the SNMP and SNMP Servers" section in the RMS Installation Guide and use the configuresnmpservingnode.sh script for setting up the trap destination. PNR Syslog Messages This table lists the Prime Network Registrar (PNR) syslog messages generated as output for the PNR alarms. Table 8: PNR Syslog Messages Conditions Syslog Messages Indicates that the PNR watchdog/main process "cnrseragt " has successfully exited. For example, Aug 6 11:55:44 dkukrety-lnx NR.cnrservagt [10263]: Info: Network Registrar 8.3 build # Aug 6 11:55:44 dkukrety-lnx NR.cnrservagt[10263]: Info: CNR server agent startup, local mode.. <Date-time> <hostname> NR.cnrservagt[8524]: Info: server agent exiting. <Date-time> <hostname> NR.cnrservagt[8524]: Info: server agent exiting. Indicates that the PNR watchdog/main process "cnrservagt" has started on the host. For example, <Date-time> <hostname> NR.dns[10267]: Info: name/dns (Domain Name Server Release Build) startup <Date-time> <hostname> NR.cnrservagt[8210]: Info: CNR server agent startup, local mode BAC Alarms This table lists the Broadband Access Center (BAC) alarms that are triggered during the monitoring of the processes. 17

18 BAC Alarms Table 9: BAC Alarms OID DPE process has stopped DPE process has started RDU process has stopped RDU process has started Description BAC server DPE is offline BAC server DPE is ready BAC server RDU is offline BAC server RDU is ready Note The BAC alarms carry no explicit severity information. Refer to the following objects to determine for which server the alarm is raised and the exact state of the server. Table 10: Objects Related to BAC Alarms Object OID Possible Values Description cbsstate Unknown 2 Initializing 3 Disconnected 4 Shutting Down 5 Ready Overload 6 Ready 7 Offline 8 Unlicensed Current state of the server. Values of 3, 4 and 7 can be considered Critical. A value of 6 can be considered as a Clear. Values of 1 and 8 should be considered major. A value of 2 can be considered informational. A value of 5 can be considered minor. cbsservertype DPE RDU Server to which the alarm applies. Note BAC alarms contain no explicit severity information. For information about the BAC syslog messages, see BAC Syslog Messages, on page 19. For details about configuring SNMP destinations for receiving BAC and FAP Alarms, see the "Configuring the SNMP and SNMP Servers" section in the RMS Installation Guide and use the configuresnmpservingnode.sh script for setting up the trap destination. 18

19 BAC Syslog Messages BAC Syslog Messages This table lists the Broadband Access Center (BAC) syslog messages generated as output for the BAC alarms. For more information, see BAC Alarms, on page 17. Table 11: BAC Syslog Messages Condition Notifies that the DPE hard drive is full Indicates that the DPE process has run out of memory Indicates that the RDU cannot be contacted Indicates that the DPE is connected to the RDU Indicates that process watchdog failed to start the DPE process Process watchdog detects that the DPE process unexpectedly terminated Indicates that process watchdog failed to start the DPE process Indicates that the DPE process is stopped Indicates that the DPE watchdog process bpr_agent is not running on the host Indicates that the DPE watchdog process "bpr_agent" is running on the host Indicates that RDU is out of disk space Indicates that RDU is out of memory Indicates that RDU is running out of licenses Indicates that process watchdog failed to start the RDU process Triggered when the process watchdog detects the RDU process unexpectedly terminated Indicates that process watchdog failed to start the RDU process Syslog Messages "DPE out of disk space" "DPE out of memory" "Failed to connect to RDU" "Connected to RDU" "Failed to start DPE process" "DPE process terminated" "DPE process has started" "DPE process has stopped" "Process watchdog is not running (DPE)" "Process watchdog is running (DPE)" "RDU out of disk space" "RDU out of memory" "RDU is running out of licenses" "Failed to start RDU process" "RDU process terminated' "RDU process has started" 19

20 for AP Indicates that the RDU process is stopped Indicates that the RDU process is stopped This alarm is triggered when the process watchdog detects that the webserver which runs PMG and UI services has unexpectedly terminated This alarm indicates that the process watchdog failed to start the web server which runs PMG and UI services This alarm indicates that the web server which runs PMG and User Equipment (UE) is stopped. "RDU process has stopped" "RDU process has stopped" "The tomcat unexpectedly terminated" "The tomcat has started" "The tomcat has stopped" for AP The AP Unreachable Alarm is raised when the AP goes down abruptly and breaks the connection between the FAP and Cisco RMS. To enable this alarm, see the "AP Unreachable Alarm" section of the Cisco RAN Management System Administration Guide. Whenever location verification for a particular small cell (AP) device fails, Cisco RMS raises an AP Location Verification Failure Alarm for that device. To enable this alarm, see the "AP Location Verification Alarm" section of the Cisco RAN Management System Administration Guide. The following table lists the AP alarms. OID Description AP Location Verification Failure AP device location verification has failed, its location is considered invalid, and the device is not activated by RMS Major (2) AP Unreachable AP is not reachable from Cisco RMS and is not responsive to the connection request Major (2) To view a clear trap for either of these notifications, look for the clear value of "6" in the Object cenalarmdefinition which has OID of