Network Diagnosis and Troubleshooting Summary by Bob Chan

Size: px

Start display at page:

Download "Network Diagnosis and Troubleshooting Summary by Bob Chan"

Wilfrid Wilkerson
5 years ago
Views:

1 Network Diagnosis and Troubleshooting Summary by Bob Chan Documentation Baselining Objective Discover the true performance of the network Provide comparison between normal and abnormal situations Verify policies Identify over-utilization and under-utilization areas Long-term performance and capacity prediction Steps of baselining Planning for the first baseline Start with data points which represent defined policies Collect data for day or two before actual baseline to determine whether the right data is collected from right devices Conduct network baselining on regular basis Speed up fault isolation Understand how the network affected by changes Identifying devices and ports of interest More clear report Either keep from change or change informing manner Use port description field to track the ports Determine the duration of baseline At least 7 days, 2 4 weeks is adequate Network documentation Overview Facilitate more effective troubleshooting Save time to build network configurations again Network configuration table Contain accurate and up-to-date records of components of the network. Provide information to identify and correct faults Should include: type, model, hostname, location, data link layer address, network layer address, other physical aspects Table for budgetary purpose should be separated Network topology diagrams Notations and symbols should be consistent

2 Cloud symbol = out of scope network Should include: device name, interface name, IP address, routing protocols Discover network configuration information show version device name, model, OS version (all) show ip interfaces active interfaces + addresses (R) show ip interfaces brief brief summary of interfaces (R) show ip interface {interface-name} MAC address (R) show ip protocols routing protocols enabled (R) show spanning tree/spantree spanning tree status (all) show cdp neighbors directly connected Cisco devices (all) show cdp entry {device id} details of connected devices (all) show interfaces description active ports + addresses (S) show interfaces status ports summary (S) show etherchannel summary EtherChannel (S) show interfaces trunk Trunk ports (S) show tech-support all information (many than needed) End system configuration table End systems are important, can affect network performance Provide complete picture of the network Should include: device name, OS, IP address, subnet mask, default gateway, DNS server, high-bandwidth network applications End system topology diagrams Should include: device name, OS, IP address, subnet mask, interface names, VLANs Discover end system configuration information OS and hardware information Access command line ipconfig / winipcfg / ifconfig - TCP/IP setting route print active routes arp a ARP information ping check connectivity tracert / traceroute view routes Documentation guidelines Determine scope Know the objective Be consistent Keep the documents accessible Maintain the documentation

3 Troubleshooting methodologies and tools Overview Systematic approach can make troubleshooting manageable, less confuse and less time wasting Rocket scientist approach (theorist) Analyze until identify root cause, then correct with precision Time wasting, resources demanding Caveman approach (practical) Swap the things until the network functions again Not reliable, root cause may still present General troubleshooting process Remarks: stages are not mutually exclusive, policies should be established in each stage Step 1 Gather symptoms From alerts from NM systems, console message and users Break down the problems to smaller ones Questioning technique Ask questions which related to the problem Use each question to eliminate or discover possibilities Make the question understandable by users Ask the time of the problem first seen Ask user to recreate the problem if possible Determine the event sequence before the problem happened Match the symptoms with common problem causes Step 2 Isolate the problem Use the layer models to categorize the problems Further gather and document symptoms Step 3 Correct the problem Implement Test Document (especially a new problem is made) Approaches Types Bottom-up Work up through OSI layer model Good to deal with physical problems Check every device and document all conclusions and possibilities after obtain authorization

Top-down Work down through OSI layer model Good to deal with application problems Check every network applications and document all conclusions and possibilities after obtain authorization Divide and

4 Top-down Work down through OSI layer model Good to deal with application problems Check every network applications and document all conclusions and possibilities after obtain authorization Divide and conquer Work directly on a particular layer, based on troubleshooter s experience and symptoms If a layer is functioning, normally underneath layers are working too Selecting guidelines Tools Network management system frameworks End stations can send alerts when problems are recognized Management entities are programmed to react Agent in end stations gather information Such information will be sent via NM protocols like SNMP Five areas: Performance, Configuration, Accounting, Fault and Security Knowledge base tools - databases Performance measurement and reporting tools - Cisco view, Netsys baseliner Event and fault management tools Cisco Network Analysis Module, protocol analyzers, pair / cable testers

5 OSI layer 1 troubleshooting Critical characteristics As physical layer failed, upper layers cannot operate too Ping timeout Not able to telnet Not able to access network drives and servers Page cannot be displayed when attempting to access web pages Noncritical characteristics Equipment indicators System LED - It shows whether the system is receiving power and functioning correctly POST off = running, green = success, amber = failed Remote Power Supply (RPS) LED - It indicates whether or not the remote power supply is in use Port Mode LED - It indicates the current state of the Mode button.

6 Port Status LED - They have different meanings, depending on the current value of the Mode LED. Console messages show interfaces no keepalive pretend interface up, should not be used

Performance lower than baseline Poor configuration Incorrect clock rate, incorrect clock source, incorrect serial links (sync/async), interface shutdown, encapsulations, IP addressing, duplex and

7 Performance lower than baseline Poor configuration Incorrect clock rate, incorrect clock source, incorrect serial links (sync/async), interface shutdown, encapsulations, IP addressing, duplex and speed Inadequate capacity Unstable routing due to marginal link or port Excessive traffic across low speed link Overload server or service Exceed design limits Distance limit of cable signal attenuation Collisions Large collision domains, duplex mismatch, late collisions Use show interface ethernet/fastethernet Electromagnetic Interference (EMI) effects Impulse noise (voltage fluctuation, 270mV on 10BaseT and 30 or 40mV on 1000BaseT)), Random noise, Alien cross-talk (parallel cables) and Near End Cross Talk (untwisted cable > 13mm) Faulty media or hardware Loose cable, dirty contacts, wrong cable, return loss Power LED, Fan, power cable Resources and utilization CPU and memory

condition) Useful commands Show buffers memory buffer pool statistics Show environment power supply and temperature Show processes

8 Power Network Console (error) messages Format: %FACILITY-SEVERITY-MNEMONIC: Message-text Facility (hardware, protocol, or module) Severity (of the situation, lower number = more serious) Mnemonic (Unique identifier of the message) Message-text (describe the condition) Useful commands Show buffers memory buffer pool statistics Show environment power supply and temperature Show processes cpu/memory resources utilizations Show stacks display processor stacks, requires stack decoder Show context show exception information in NVRAM

9 OSI layer 2 troubleshooting More difficult to troubleshoot because of suboptimal operations, either frames not transmitting through best paths or dropped frames Framing errors A frame which is not ended on 8-bit byte boundary Noisy serial line Improperly designed cable Incorrect clock (rate) T1 link problem because of incorrect framing or coding specification Use show interfaces to reveal Frame error count Invalid Cyclic Redundancy Check Layer-2 to Layer-3 address mapping errors Occur in point-to-multipoint, Frame Relay and broadcast Ethernet A correct destination Layer-2 address must be given to a frame Layer-2 to Layer-3 address mapping mechanism and potential errors Static maps In Ethernet environment, change of NIC can lead to problem In Frame Relay environment, incorrect DLCIs assigned by Telco Dynamic maps (ARP) Devices do not respond to ARP or Inverse-ARP requests Invalid ARP replies due to misconfiguration, DoS or Man-inthe-middle attacks Symptoms (except man-in-the-middle attack) No direct Layer-3 communications Layer-2 communications are ok No or incorrect Layer-2 address when doing ARP inspection Useful commands Show arp Show cdp neighbor detail Show frame-relay map Spanning Tree Protocol Problem occur when exchange of Bridge Protocol Data Units (BPDUs) failed Symptoms Unusually high backplane utilization

10 Rapid address re-learning Rapidly incrementing frame counters Poor link performance Broadcast storm within Layer-2 domain Causes Bad transceivers Cabling issues Hardware failures such as ports and Supervisor engine Unidirectional link between bridges (cause STP loops) UDLD protocol (to prevent STP loops) A Layer-2 protocol which works with Layer-1 mechanisms Able to detect neighbors identity and shutdown misconnected ports Operations Exchange protocol packets between neighbors Packets contains device/port ID of device itself and of neighbors Neighboring ports should see their own echo in packets received from another side, otherwise the link will be considered as unidirectional link after specific time The ports in unidirectional link will be disabled by UDLD, and only can reenable manually Configuration UDLD is disabled by default Use udld enable either in global mode or in a particular interface (interface command overwrites global ones) Use show udld interface to verify UDLD operation Ethernet broadcast traffic Causes Poorly programmed or configured applications Huge Layer-2 broadcast domain Other network problems such as STP loops or route flapping Discover Either compare with baseline or use protocol analyzer Solutions Create separate VLANs Configure switches to be multicast aware Use scheduling for distribution services to control broadcast

11 Ethernet switch flooding Causes Asymmetric routing because of HSRP configuration on Layer-3 switches STP Topology Change Notification (TCN) Overflow of switch forwarding table (CAM) Solutions Set the router s ARP timeout and switches forwarding table-aging time close to each other Enable STP portfast feature on ports Use port security feature EtherChannel Cause Non-identical configuration on both sides Symptoms Loss of connectivity (due to switching loops) Increased backplane utilization Rapid MAC address re-learning Interfaces may turn to ErrDisable state Solution Configure the ports on both sides to have same speed, duplex, and native VLAN trunk T1 framing errors Use show controllers t1 Check if clock source is provided by Telco (Line) Check if the framing format is same as the line Check if the line coding matches ISDN Useful commands show isdn status debug isdn q931 show Layer-2 exchange debug dialer show dialer list and dialer map Check PPP connection Frame Relay Check physical connectivity Verify LMI information exchange (show frame-relay lmi) Verify PVC status (Active, inactive or deleted) Verify Frame Relay encapsulation

12 OSI layer 3 troubleshooting General Distribute-list blocking (except OSPF and ISIS) Passive interface (RIP/IGRP can still receive routing updates) Missing or incorrect network or neighbor statement Layer-1 and 2 problem show ip protocols show ip interface show ip interface brief debug ip routing RIP Incompatible version types By default, router receives version 1 and 2 but send version 1 only Mismatched authentication key (in version 2 only) Hop count limit (more than 15) Discontiguous networks Add static route Change the middle network into a part of major network also Use version 2 with no auto-summary Invalid source address Cause by IP unnumbered Use no validate-update-source to solve the problem Flapping routes Large routing table debug ip rip EIGRP Mismatched K values on both sides Default K1=1, K2=1, K3=1, K4 and K5=0 Stuck in active Congested or bad link Low router resources Long query range Excessive redundancy Duplicate router ID Change loopback address show ip eigrp interfaces show ip eigrp neighbors debug ip eigrp

13 OSPF Access list blocking (multicast hello ) Mismatched parameters Hello and dead interval Authentication type Area ID Area options State issues Stuck in ATTEMPT No response when trying to contact a neighbor Misconfigured neighbor statement Stuck in INIT Two-way communication has not been established Access list blocking OSPF hellos Authentication enabled on one side only Stuck in EXCHANGE Fail to exchange Database Descriptor (DBD) packets Duplicate router ID Mismatched interface MTU Point-to-point link unnumbered show ip ospf interface debug ip ospf events BGP Neighbors not initializing Updates will only be exchanged upon Established neighbor state Routes not being installed in routing table IBGP Routes not synchronized Next hop is unreachable EBGP Next hop is unreachable in case of multihop EBGP Multiexit discriminator (MED) value is infinite ISIS Adjacency problems Show clns neighbors debug isis adj packets debug isis update-packets

14 OSI layer 4 troubleshooting ACL Implementing the standard access list as close to the protected destination Implementing the extended access list as close as possible to the source of the traffic being filtered. show log show ip access-list {number/name} show ip interface NAT DHCP Source address of DHCP-Request packet is Since NAT requires both valid destination and source address, DHCP is difficult to run on router with NAT DNS and WINS When using dynamic NAT, the inside and outside addresses relationship changes frequently, so the outside DNS servers can not accurately present the network inside the router SNMP SNMP management station may not be able to contact SNMP agents on the other side of the NAT router because NAT can alter the addressing information in the payload show ip nat debug ip nat Others Local system logging logging on Network Time Protocol (NTP) ntp peer {NTP server IP address} ntp peer authenticate Logging timestamps Service timestamps debug datetime {local time} {msec} {show timezone} NetBIOS Netstat display protocol statistics and current TCP/IP connections Nbstat - display protocol statistics and current NetBIOS connections running on TCP/IP

Connections, addressing and common configuration rules.

Connections, addressing and common configuration rules. Lab #1 r9-1 s2 e1 3/11 VLAN_21 BGP AS 100 s1 3/1 e0 s0 e0 3/2 s1 VLAN_22 BGP AS 200 r1 r2 Ring 4 r4 s0/1 r3 to0/0 s0/0 s0 s1 OSPF Area 33 to0 e0/0 e0 OSPF Area 4 3/4 b0/0 3/3 VLAN_14 VLAN_13 403 304 c15