CHAPTER 8 Troubleshooting Revised: December 21, 2012, Introduction Your Cisco SCE 8000 GBE platform went through extensive testing before leaving the factory. However, if you encounter problems starting it, use the information in this chapter to help isolate the cause of the problems. The procedures in this chapter assume that you are troubleshooting the initial system startup, and that your Cisco SCE 8000 GBE platform is in the original factory configuration. If you have removed or replaced components or changed any default settings, the recommendations in this chapter might not apply. Make sure to review the safety warnings listed in the Regulatory Compliance and Safety Information for the Cisco SCE8000 GBE document that accompanied your Cisco SCE 8000 GBE platform before using the troubleshooting procedures in this chapter. Troubleshooting Overview, page 8-2 Troubleshooting with the User Log, page 8-13 8-1
Troubleshooting Overview Chapter 8 Troubleshooting Troubleshooting Overview This section describes the troubleshooting methods used in this chapter and describes how the Cisco SCE 8000 GBE platform is divided into subsystems for more efficient problem solving. If you are unable to easily solve the problem, contact a customer service representative for assistance and further instructions. Provide the representative with the following information: Date you received the Cisco SCE 8000 GBE Chassis serial number Type of software and release number Brief description of the problem you are having Brief explanation of the steps you have taken to isolate and resolve the problem Maintenance agreement or warranty information Table 8-1 shows the general troubleshooting strategy described in this chapter. See this table, as necessary, to follow the steps to isolate problems to a specific subsystem and resolve the problem if possible. Table 8-1 Troubleshooting Strategy for Startup Problems Action Yes No Step 1 Turn power on. Go to Step 2. Step 2 Check the following: Front panel power LED on? Power supply 'Input OK' LEDs on? 'Output fail' LEDs not on? Go to Step 3. Step 3 Status LED red (failure) See the Troubleshooting the Firmware Package Installation section on page 8-8 and go to Step 4. See the Troubleshooting the Power Subsystem section on page 8-8 and go to Step 3. Go to Step 4. Step 4 Management interface operational? Go to Step 5. See the Troubleshooting the Management Subsystem section on page 8-9 and go to Step 5. Step 5 Link interfaces operational? Go to Step 6. See the Troubleshooting the Link Interface Subsystem section on page 8-11 and go to Step 6. Step 6 System startup successful (all interfaces operating normally). Information About Troubleshooting Tools There are two tools that can help you to successfully troubleshoot your Cisco SCE 8000 GBE installation: CLI Commands for Troubleshooting, page 8-3 Checking the LEDs, page 8-5 8-2
Chapter 8 Troubleshooting Information About Troubleshooting Tools CLI Commands for Troubleshooting Use the following commands to provide information to help you troubleshoot the installation of your Cisco SCE 8000 GBE platform. See the Cisco SCE8000 GBE Software Configuration Guide or the Cisco SCE8000 CLI Command Reference for more information. Note Remember that if the management interface is not operational, you should connect the Cisco SCE 8000 GBE platform to a local console so that you can enter CLI commands for troubleshooting. Troubleshooting firmware package installation: boot system <filename> Specifies and verifies the package file to be installed. Error messages or other output identify problems with the package file. Following is a sample output from the boot system command: SCE(config)# boot system ftp://cisco:cisco@10.10.10.10/downloads/senum.pkg.pkg Verifying package file SENum.pkg.pkg Package file verified OK. Troubleshooting the management subsystem: show interface GigabitEthernet 1/1 Displays IP address and auto-negotiation information for the management interfaces. Following is a sample output from the show interface GigabitEthernet 1/1 command: ip address: 10.1.6.145 subnet mask: 255.255.0.0 Configured speed: auto, configured duplex: auto AutoNegotiation is On, link is Up, actual speed: 100, actual duplex: half show ip default-gateway Displays the IP address of the configured default gateway. Following is a sample output from the show ip default-gateway command: Default gateway: 10.1.1.1 show ip route Displays the entire routing table and the destination of last resort (default-gateway). Following is a sample output from the show ip route command: gateway of last resort is 10.1.1.1 show access-lists Shows all access-lists or a specific access list. Following is a sample output from the show access-lists command: Standard IP access list 1 Permit 10.1.1.0, wildcard bits 0.0.0.255 deny any show telnet Displays the status of the telnet server daemon (status) or any active Telnet sessions (sessions). 8-3
Information About Troubleshooting Tools Chapter 8 Troubleshooting Following is a sample output from the show telnet command: show telnet sessions There is 1 active telnet session: Index Source ================ 0 10.1.1.201 show telnet status Telnet deamon is enabled. show line vty timeout Shows the timeout configured for Telnet sessions. Following is a sample output from the show line vty timeout command: Timeout is 30 minutes Troubleshooting the link interface subsystem: show interface GigabitEthernet 3/#/# Displays information for a specific GBE Interface. Following is a sample output from the show interface command: Auto negotiation configured: Disabled Actual status: Link is: ON Auto negotiation: Disabled Bandwidth (L1): 10000000 Kbps, Burst-size: 500000 bytes Pseudo IP Address: Not Configured show interface GigabitEthernet 3/#/# counters Displays the values of counters of a GBE interface. Following is a sample output from the show interface counters command: L2 In total octets: 792000 In good unicast packets: 12000 In good multicast packets: 0 In good broadcast packets: 0 In packets discarded: 0 In packets with CRC/Alignment error: 0 In undersized packets: 0 In oversized packets: 0 Rx pause packets: 0 L2 Out total octets: 0 Out unicast packets: 0 Out good multicast packets: 0 Out good broadcast packets: 0 Out packets discarded: 0 Tx pause packets: 0 Tx regular collision events: 0 L2 Bandwidth Kbps (Rx + Tx): 0 # of packets received of length (in octets): 64: 0, 65-127: 12000, 128-255: 0, 256-511: 0, 512-1023: 0, 1024-1518: 0, 1519+: 0 See the Troubleshooting with the User Log section on page 8-13 for an explanation of commands related to the user log. 8-4
Chapter 8 Troubleshooting Information About Troubleshooting Tools Checking the LEDs The LEDS on the Cisco SCE 8000-SMC-E front panel, along with the LEDs on the power supplies and fan assembly are the most immediate problem-detection mechanism of the platform. See the following sections for information on Cisco SCE 8000 GBE platform LEDS: Table 2-3 on page 2-4 Examining the LEDs, page 6-14 Starting the System and Observing Initial Conditions, page 7-3 Cisco SCE 8000 GBE Operational Status, page 8-5 Cisco SCE 8000 GBE Operational Status Table 8-2 lists the operational states of the Cisco SCE 8000 GBE. The Status LED on the Service Control module reflects the current Cisco SCE 8000 GBE operational status (See Table 8-3 and Table 8-4). Once boot is complete, the operational status can be displayed using the show system operation-status CLI command. Table 8-2 Cisco SCE 8000 GBE Operational States Cisco SCE 8000 GBE Operational Status Description Status LED State Booting Initial state after reset Amber Operational Cisco SCE 8000 GBE becomes operational after completing the following process: Steady green Boot is completed Power-on self-tests are completed without failure Platform configuration is applied 8-5
Information About Troubleshooting Tools Chapter 8 Troubleshooting Table 8-2 Cisco SCE 8000 GBE Operational States (continued) Cisco SCE 8000 GBE Operational Status Description Status LED State Warning Failure Cisco SCE 8000 GBE is fully operational (as above) but one of the following occurred: GBE Management port link is down Internal temperature above threshold Internal voltage not in expected range Fan problem Power supply problem Insufficient space on the disk Note If the condition that caused the Cisco SCE 8000 GBE to be in Warning state is resolved (for example, link is up) the Cisco SCE 8000 GBE reverts to Operational state. System is in Failure state after Boot because of one of the following conditions: Power-on test failure Three abnormal reboots in less than 30 minutes Platform configured to enter Failure mode consequent to failure-induced reboot (this is configurable using CLI command) Severe system health problem, such as extensive overheating or voltage out of correct operating range. Note Depending on the cause of failure, the management interface and the platform configuration may or may not be active/available. Flashing amber Red Table 8-3 Power Supply LEDs LED Label Color State Function INPUT OK Green On The input voltage is present and within the required range. Off The input voltage is not present or not within the required range. OUTPUT FAIL Green On The output voltage is not within the required range. Off The output voltage is in the required range. FAN OK Green On Power supply internal fan is operational. Off Power supply internal fan is not operational. Power (front panel) Green Steady Installed power supplies are functioning normally. 8-6
Chapter 8 Troubleshooting Problem Solving Using a Subsystems Approach Table 8-3 Power Supply LEDs (continued) LED Label Color State Function Amber Steady One of the power supply units is disconnected or malfunctioning. Off No power. Table 8-4 Fan Assembly LED LED Label Color State Function FAN STATUS Green On All fans are operational. Off One or more of the individual fans are not operational. Problem Solving Using a Subsystems Approach Identifying Startup Problems Identifying Startup Problems, page 8-7 Troubleshooting the Power Subsystem, page 8-8 Troubleshooting the Firmware Package Installation, page 8-8 Troubleshooting the Management Subsystem, page 8-9 Troubleshooting the Link Interface Subsystem, page 8-11 Startup problems are commonly because of the source power or a poor cable connection. When you start up the Cisco SCE 8000 GBE platform for the first time, you should observe the startup sequence described in Starting the Cisco SCE 8000 GBE Platform section on page 7-2. This section contains a more detailed description of the normal startup sequence and describes the steps to take if the system does not perform that sequence as expected. LEDs indicate all system states in the startup sequence. By checking the state of the LEDs, you can determine when and where the system failed in the startup sequence. Use the following descriptions to isolate the problem to a subsystem, and then proceed to the appropriate sections to try to resolve the problem. When you start up the system by turning on the power supply switch, the following should occur: You should immediately hear the fans operating. If the Status LED is flashing orange, indicating a warning state, check the user log. At the prompt, enter more user log. If any of the following warning messages appear, and the root cause is not obvious and easily solved (such as obstruction of external air-flow) turn the Cisco SCE 8000 GBE platform off and call technical support: voltage problem fans problem abnormal raise in interior temperature 8-7
Problem Solving Using a Subsystems Approach Chapter 8 Troubleshooting Troubleshooting the Power Subsystem Check Table 8-5 to help isolate a problem in the power subsystem. In the normally configured Cisco SCE 8000 GBE platform with redundant power supply units, it is unlikely that the device does not start at all. However, at startup it should be verified that both power supply units are operational, and therefore the following steps should be followed if the Power LED on the front panel remains unlit when the Cisco SCE 8000 GBE platform is powered up. Note If the system powers off because of an environmental shutdown, wait at least one minute before manually rebooting the system, or it pauses indefinitely. Table 8-5 Troubleshooting the Power Subsystem Symptom Possible Cause Possible Solution Power LED on the front panel and LEDs on the power supply unit are not lit, or do not remain lit continuously. Power cable not fully seated at system. Troubleshooting the Firmware Package Installation Turn the power switch to the off position and reseat the power cable in the system. Power cable not fully seated at source. Turn the switch to the off position and reseat the power cable at the power source. Power source is faulty. Turn the switch to the off position, connect the power cable to another power source, if available, and turn the switch back on. Faulty power cable. Turn the switch to the off position, remove the cable and replace it. Faulty power supply. If the system still fails to come up when the power supply is connected to a different power source with a new power cable, the power supply unit is probably faulty. Contact a service representative. Check Table 8-6 to help isolate a problem in the installation of the firmware package. Problems related to the installation of the firmware package could be any of the following: File not found in the expected location. Wrong file type. Device to which the file is to be extracted is full. 8-8
Chapter 8 Troubleshooting Problem Solving Using a Subsystems Approach Table 8-6 Troubleshooting the Firmware Package Installation Diagnostic Action Enter the CLI command: configure boot system <filename> Symptom Possible Cause Possible Solution Returned error is: The package file does not exist in the specified location. Error-File <filename> does not exist In the output of the command, the package file platform is not the correct installation file for the Cisco SCE 8000 GBE. Package file platform mismatch. Troubleshooting the Management Subsystem Check Table 8-7 to help isolate a problem in the management subsystem. Problems in the management subsystem could be any of the following: Management link is down (Mng LINK LED not lit also Status is WARNING) Management link is up (Mng LINK LED lit) but does not answer ping Telnet connection cannot be established because of link problems (Mng LINK LED not lit) Management link is up (Mng LINK LED lit) but Telnet connection cannot be established Telnet connection established, but terminates automatically Verify package file location and try again. Verify that you have the package file appropriate to your platform type. Note When the management link is down and or or a Telnet connection cannot be established, you must open a CLI session on a local terminal connected to the CON port. This enables you to solve the problem and then reconnect through the management port. Table 8-7 Troubleshooting the Management Subsystem Symptom Diagnostic Action Possible Cause Possible Solution Management link down: CLI command show RJ 45 connector is not Reconnect the cable to the interface connected to the platform or GBE port and to network. Mng LINK LED not lit GigabitEthernet 1/1 to the network. ping to management interface fails Cable is broken. Check or replace the cable. 8-9
Problem Solving Using a Subsystems Approach Chapter 8 Troubleshooting Table 8-7 Troubleshooting the Management Subsystem (continued) Symptom Diagnostic Action Possible Cause Possible Solution Management link up: CLI commands One of the following configurations may be wrong: Mng LINK LED is lit ping to management interface fails show ip route show ip default-gateway CLI command Telnet connection cannot be established Mng LINK LED is not lit (link is down) Telnet connection cannot be established Mng LINK LED is lit (link is up) show access-lists CLI command show interface GigabitEthernet 1/1 CLI command: show telnet status CLI command: show telnet sessions CLI command: show ip default-gateway IP address / subnet mask IP default gateway An ACL may be assigned that denies entry. Management interface IP address or subnet mask is incorrect. Telnet server is disabled. Too many Telnet connections (up to five concurrent sessions are supported via Telnet). Default gateway is incorrect (when the host used as client is not in the same network as the Cisco SCE Platform). See Initial Setup Parameters section on page 5-3. See the Setting the IP Address and Subnet Mask of the Management Interface section in the Cisco SCE8000 GBE Software Configuration Guide. See Initial Setup Parameters section on page 5-3. See the Configuring Access Control Lists (ACLs) section in the Cisco SCE8000 GBE Software Configuration Guide. Check or reconfigure management port IP address and subnet mask. Enable Telnet server: service telnetd Close one or more of the open Telnet sessions. Check or reconfigure default gateway. See Initial Setup Parameters section on page 5-3. See the Setting the IP Address and Subnet Mask of the Management Interface section in the Cisco SCE8000 GBE Software Configuration Guide. 8-10
Chapter 8 Troubleshooting Problem Solving Using a Subsystems Approach Table 8-7 Troubleshooting the Management Subsystem (continued) Symptom Diagnostic Action Possible Cause Possible Solution CLI command: show ip route <host-ip-address> CLI commands: show access-lists show ip access-class Telnet connection terminates automatically CLI commands: show line show line vty timeout Troubleshooting the Link Interface Subsystem Routing tables are incorrectly configured (when the host used as client is not in the same network as the Cisco SCE Platform, and there is more than one gateway on the Cisco SCE Platform network). Host is not a member of a valid access-list. Telnet connection may be timing out. Check or reconfigure routing tables. See Initial Setup Parameters section on page 5-3. See the Setting the IP Address and Subnet Mask of the Management Interface section in the Cisco SCE8000 GBE Software Configuration Guide. See Initial Setup Parameters section on page 5-3. See the Configuring Access Control Lists (ACLs) section in the Cisco SCE8000 GBE Software Configuration Guide. Reconfigure line timeout. timeout <time in seconds> Check Table 8-8 to help isolate a problem in the link interface subsystem. In general, the case where no traffic is coming out of the Cisco SCE 8000 GBE is often caused by link problems or the GBE traffic interface configuration. Note that in some cases, the problem which seems like a sending problem could be in the Rx (no traffic is being received by the Cisco SCE 8000 GBE or there is actually no traffic on the line, which could be a normal situation). Note In CLI commands for GigabitEthernet interfaces, /#/# stands for the number of the SPA module (Cisco SCE 8000-SIP subslot) (0-3) followed by the port number (0 to 7). Problems in the link interface subsystem could be any of the following: Link is down (LINK LED not lit and system status is WARNING) Peer does not receive traffic from Cisco SCE 8000 GBE (LINK LED is lit and Tx LED is flashing) 10GBE link is up but not receiving from peer (LINK LED is lit, but Rx LED is not flashing) 8-11
Problem Solving Using a Subsystems Approach Chapter 8 Troubleshooting Table 8-8 Troubleshooting the Link Interface Subsystem Symptom Diagnostic Action Possible Cause Possible Solution Link is down. (LINK LED not lit) Output counters not incrementing. Downgrading Cisco SCOS CLI command: show interface GigabitEthernet 3/#/# counters Connector is not connected to the platform or to the network. While downgrading Cisco SCOS Release 3.8.x to an earlier version, an error message similar to the following is displayed: "Could not allocate 7383087 bytes for Control Application XML data partition" To downgrade Cisco SCOS Release 3.8.x to an earlier version, follow these steps: Reconnect the fiber to the GBE port and to network. Fiber is broken or damaged. Reconnect or replace the fiber to the GBE port. Connectivity using external optic bypass is incorrect or problematic. Temporarily disconnect optic bypass module and check operation. See Cabling the GBE Line Interface Ports section on page 6-8. GBE link is up (LINK LED is continuous green) No traffic received (GBE interface Rx LED is not flashing) Problem with external optic bypass module. No traffic is being sent to the Cisco SCE 8000 GBE from its peers. Reconnect or replace the fiber between the GBE port and the optic bypass module. Replace the optic bypass module. Check traffic connection at peer. Step 1 Step 2 Step 3 Step 4 Step 5 Copy the Cisco SCOS Release 3.8.x PQI file to the /apps/data/scos/app/ folder: SCE8000#> copy ftp://ftpdefaultdirectory/3.8.x/<filename.pqi> /apps/data/scos/app/ Uninstall the Cisco SCOS Release 3.8.x PQI file: SCE8000(config if)#>pqi uninstall file <filename.pqi> Copy the running configuration to the startup configuration: SCE8000#> copy running-config startup-config Install the desired version of the Cisco SCOS PKG file from the Cisco SCA BB application. Install the desired version of the Cisco SCOS PQI file from the Cisco SCA BB application. 8-12
Chapter 8 Troubleshooting Troubleshooting with the User Log Troubleshooting with the User Log The user log is an ASCII file that can be viewed in any editor. It contains a record of system events, including startup, shutdown and errors. You can use the Logger to view the user log to determine whether or not the system is functioning properly, as well as for technical support purposes. Logging System, page 8-13 Generating a File for Technical Support, page 8-14 Logging System Events are logged to one of two log files. After a file reaches maximum capacity, the events logged in that file are then temporarily archived. New events are then automatically logged to the alternate log file. When the second log file reaches maximum capacity, the system then reverts to logging events to the first log file, thus overwriting the temporarily archived information stored in that file. Basic operations include: How to Copy the User Log to an External Source, page 8-13 How to Copy the User Log to an Internal Location, page 8-13 How to View the User Log, page 8-13 How to Clear the User Log, page 8-14 How to View the User Log Counters, page 8-14 How to View the Non-volatile Counter For the User-file-log Only, page 8-14 How to Copy the User Log to an External Source You can view the log file by copying it to an external source. This command copies both log files to any external host running a FTP server. From the SCE# prompt, enter logger get user-log file-name ftp://username:password@ipaddress/path and press Enter. How to Copy the User Log to an Internal Location How to View the User Log You can view the log file by copying it to disk. This command copies both log files to the local Cisco SCE platform disk. From the SCE# prompt, enter logger get user-log file-name target-filename and press Enter. Note This command is not recommended when the user log is large. Copy a large log to a file to view it (see How to Copy the User Log to an External Source section on page 8-13). From the SCE# prompt, enter more user-log and press Enter. 8-13
Generating a File for Technical Support Chapter 8 Troubleshooting How to Clear the User Log You can clear the contents of the user log at any time. The user log contains important information regarding the functioning of the system. It is recommended that a copy be made before the log is cleared. Step 1 Step 2 Step 3 From the SCE# prompt, enter clear logger device user-file-log and press Enter. The system asks Are you sure? Enter y and press Enter. How to View the User Log Counters There are two types of log counters: User log counters count the number of system events logged from the Cisco SCE platform last reboot Non-volatile counters are not cleared during boot time From the SCE> prompt, enter show logger device user-file-log counters and press Enter. How to View the Non-volatile Counter For the User-file-log Only From the SCE> prompt, enter show logger device user-file-log nv-counters and press Enter. Generating a File for Technical Support For technical support to be most effective, the user should provide them with the information contained in the system logs. Use the logger get support-file command to generate a support file for the use of Cisco technical support staff. Step 1 Step 2 From the SCE# prompt, enter logger get support-file filename and press Enter. The support information file is created using the specified filename. This operation may take some time. To copy the support file to an external source, from the SCE# prompt, enter copy filename ftp://username:password@ipaddress/path and press Enter. 8-14