Root-Cause Network Troubleshooting Optimizing the Process Tim Titus CTO, PathSolutions

Transcription

1 Root-Cause Network Troubleshooting Optimizing the Process Tim Titus CTO, PathSolutions 1

2 Agenda Business disconnect Why is troubleshooting so hard? Troubleshooting methodology Tool selection Finding the root-cause Achieving Total Network Visibility 2

3 Business Disconnect You re responsible for the entire network Most network engineers know less about their network s health and performance than their user community You can t manage what you can t measure -- Peter Drucker 3

4 4 Don t turtle your network

5 Why is Troubleshooting so Hard? Business Reasons Networks are getting more complex Less staff remains to support the network Technical Reasons Proper methodology is not utilized Wrong tools are employed 5

6 Troubleshooting Methodologies What graduates a junior level Engineer to a senior level Engineer is their troubleshooting methodology 6

7 Bad Methodology Do something to try to fix the problem Reboot the device Change the network settings Replace hardware Re-install the OS 7

8 Good Methodology Collect information Create hypothesis Test hypothesis Undo changes Implement fix Verify Original Problem is Solved and no new problems exist Notify users Document fix 8

9 Tool Selection Types of Tools Cable Testers Packet analyzers/capture Application Performance Monitoring (APM) Flow collectors SNMP Collectors 9

10 Cable Testers Using a cable tester to solve a call quality problem Results 4.3db of loss NEXT detected Cable Tester x41 A G x51 B D F H x42 C E Actual VoIP Call I x52 x43 You have information about Layer 1 on one link in the network x53 10

11 Cable Testers Good for: Confirming physical issues on one link in the network Bad for: Finding physical issues on the network Determining application usage Finding bandwidth limitations Finding device limitations 11

12 Packet Capture Using a sniffer to solve a call quality problem Results of VoIP Call Latency: 127ms Jitter: 87ms Packet loss: 8.2% Packet Capture x41 A G x51 B D F H x42 C E Actual VoIP Call I x52 x43 You have confirmation that there is a problem, but no idea which device or link caused the packet loss x53 12

13 Packet Capture Good for: Confirming packet loss (Are we missing packets?) Confirming packet contents issues (No QoS tagging on packets when there should be) Determining application-level issues (Source and destination IP and ports used for a session) Bad for: Finding physical, data-link, or network issues Finding bandwidth limitations Finding device limitations 13

14 Application Performance Monitoring Using APM to determine performance through the network Results of Simulation Latency: 127ms Jitter: 87ms Packet loss: 8.2% Agent x41 A G x51 B D F H x42 C E Simulated VoIP Call I x52 x43 Agent You have knowledge of the experience across the network, but no understanding of the source or cause of the problem. x53 14

15 Application Performance Monitoring Good for: Measuring user experience across the network (Are we having problems right now?) Bad for: Finding physical, data-link, or network issues Finding bandwidth limitations Finding device limitations 15

16 Flow Collectors Using a flow collector to determine usage of the network x41 A Results of Flow SourceIP: :80 DestinationIP: :3411 Packets: 251 Bytes: 19,386 Flow Collector G x51 B D F H x42 C E Actual VoIP Call I x52 x43 You have knowledge of a transfer across the network, but no recognition if there were any problems with the transfer. x53 16

17 Flow Collectors Good for: Determining communications across the network Who is using a link? When do they use it? What do they use it for? Bad for: Finding physical, data-link, or network issues Finding bandwidth limitations Finding device limitations 17

18 SNMP Collectors Collecting information from switches and routers to discover faults Results of Collection WAN link is overloaded at 2:35pm SNMP Collector x41 A G x51 B D F H x42 C E Actual VoIP Call I x52 x43 You have data about conditions on some parts of the network, but no analysis of the problem or correlation to events x53 18

19 SNMP Collectors Good for: Tracking packet loss per interface/device (Are we dropping packets on a link? why?) Monitoring device and link resource limitations (Are we over-utilizing a link? Is the router CPU pegged?) Bad for: Determining who is using the network Finding application layer problems 19

20 Finding the Root-Cause x41 A Poor Quality VoIP Call G x51 B D F H x42 x43 C Step 1: Identify the involved endpoints and where they are connected into the network E I x52 x53 20

21 Finding the Root-Cause x41 A G x51 B D F H x42 x43 C E Step 2: Identify the full layer-2 path through the network from the first phone to the second phone I x52 x53 21

22 Finding the Root-Cause x41 A G x51 B D F H x42 x43 C Step 3: Investigate involved switch and router health (CPU & Memory) for acceptable levels E I x52 x53 22

23 Finding the Root-Cause x41 A TRANSIENT PROBLEM WARNING: If the error condition is no longer occurring when this investigation is performed, you may not catch the problem G x51 B D F H x42 C E Step 4: Investigate involved interfaces for: I x52 x43 VLAN assignment DiffServe/QoS tagging Queuing configuration 802.1p Priority settings Duplex mismatches Cable faults Half-duplex operation Broadcast storms Incorrect speed settings Over-subscription x53 23

24 Optimizing the Methodology In a perfect world, you want: Monitoring of: Every switch, router, and link in the entire infrastructure All error counters on the interfaces QoS configuration and performance Continuous collection of information Automatic layer-1, 2, and 3 mapping from any IP endpoint to any other IP endpoint Problems identified in plain-english for rapid remediation This is what PathSolutions TotalView does 24

25 Deployment All Switches and Routers are queried Install TotalView for information Result: One location is able to monitor all devices and links in the entire network for performance and errors 25

26 Total Network Visibility Broad: All ports on all routers & switches Continuous: Health collected every 5 minutes Deep: 18 different error counters collected and analyzed Network Prescription engine provides plain- English descriptions of errors: This interface is dropping 12% of its packets due to a cable fault 26

27 Results Within 12 Minutes Establish Baseline of Network Health 12% Loss from Alignment Errors 28% Loss from Duplex mismatch 7% Loss from cabling fault 11% Loss from Jumbo Frame Misconfiguration 27

28 Results Within 12 Minutes Repair Issues 12% Loss from Alignment Errors 28% Loss from Duplex mismatch 7% Loss from cabling fault 11% Loss from Jumbo Frame Misconfiguration 28

29 Path Analysis Report 7:56am 18% Loss from FCS Errors 12:02pm 12% Loss from Collisions 11:32am 100% Transmit utilization 15% Loss from discards Latency & Jitter penalty incurred 29

30 30 Demo

31 Take the TotalView Tour to get a Free Network Equipment Magnet Set With it, you will always have an easy way to map out your network on any white board! Visit us at Booth #

32 32