Trace Collection Guidelines WiNG 5 Vik Evans Systems Engineer Enterprise Networking and Communications 1
Troubleshooting Checklist Mandatory Information ID Description Response 1 Customer 2 Perceived Problem 3 Problem identified by tier-ii, including underlying issues. 4 Config files for devices, switches, AP s and firmware versions 5 Steps used to reproduce problem on test bench 6 Obtain Syslogs 7 Obtain appropriate wireless & wired traces; Aeropeek / Omnipeek 8 Network topology (logical / physical layout) 2
Troubleshooting Checklist (cont.) Useful Information ID Description Response 1 Customer Disposition 2 Duration of problem thus far 3 Severity of impact on operations 4 Current work-arounds customer may be using 5 Does the problem occur at multiple sites? 6 Can the customer reproduce the problem? 3
Troubleshooting Checklist (cont.) Optional Information (based on relevance) ID Description Response 1 Intermittent? If so, what is the frequency? 2 Any changes to network / configurations recently? 3 Additional configuration information for AP s 4 Multiple ESS s in use? 5 Trunking? 6 Security info on WLAN(s) 7 Bluetooth Enabled? 8 Network Addresses (MAC & IP) 9 Mobile device types (scanners / VoIP phones / laptops) 10 Vendors / Models 11 Did customer have a site survey performed? 12 Is there proper cell density for coverage? 13 Are there known areas of poor coverage? 14 Is switch redundancy being used? 15 Can engineer(s) visit site? 16 Environmental temperature of site? 4
SPR Pre-Requisite: Trace Files This presentation focuses on obtaining relevant trace files, prior to opening an SPR. It will cover traditional methods, using laptop software as well as the capabilities within WiNG 5. 5
Trace Collection Guideline Objective: Minimize problem identification by providing guidelines on collection of traces and doing a basic analysis to ensure all the relevant information has been captured Agenda: Overview of Trace Collection Tools Trace Collection Procedures Basic Trace Analysis 6
Section I: Packet Analysis Utilities 7
Trace Collection & Analysis Utilities WildPackets OmniPeek (formerly AiroPeek) Netscout Sniffer Analysis (formerly Network General Sniffer Pro) Wireshark Other Wireless Analysis Utilities Riverbed / Cace Technologies - WifiPilot AirMagnet ECRT uses Wildpackets OmniPeek and Wireshark as the standard capture analysis tools. OmniPeek and Wireshark will be the focus of this presentation. 8
Host Setup Ensure proper drivers / libraries are installed for your host device. Wildpackets has many device drivers at: http://www.wildpackets.com/support/omni/omnipeek_ent erprise/wireless Wireshark will depend on adapter and host OS: *nix OS will use the libpcap library, included Windows OS will us winpcap, which will install during Wireshark installation AirPcap is a link-layer library and adapter used to perform wireless capture in Wireshark on a Windows host 9
OmniPeek Dashboards In Network Dashboard mode, OmniPeek displays key statistics like utilization, wireless signal and recently saved files. 10
OmniPeek Capture Window Elements Some common interface elements are shown. 11
Performing OmniPeek Captures Click the New Capture icon Click Adapter in the Capture Options dialog and select the desired capture interface. Click General, name the capture and specify continuous or not. Note that wireless capture will not be possible without a supported adapter and drivers. 12
Wireshark Wireshark (formerly Ethereal) is a free, open-source utility that, over the years, has developed into a very robust packet analysis application. Wireshark runs on many platforms, including Windows, Mac OS X and Linux. 13
Wireshark Startup Screen & Elements 14
Wireshark Notes: Wireshark is able to save in / work with formats recognized by OmniPeek, so there is no concern for incompatibilities. Any adapter that shows up in the Interface List is available for capture. Promiscuous mode will capture wireless packets of the SSID the adapter is joined to only. Monitor mode will allow capture of all 802.11 packets heard, however will not allow membership to any WLAN it is purely for capture. 15
Performing Wireshark Captures Click the Capture Options icon Select the desired adapter from the drop-down menu. At Interface you can specify whether the capture is local, or a remote-host. This would be another Wireshark machine configured to listen for incoming requests Click Start. 16
Planning and Validation Applications Other utilities exist for planning, validation and troubleshooting that should be used initially for proper implementation of a wireless network. Motorola LANPlanner Motorola AirDefense Mobile Predictive planning and site survey validation for both AirMagnet Survey Site survey / coverage validation 17
Section II: WiNG 5 Packet Capture 18
The packet capture features of WiNG 5 enable one to collect traces from almost any point in a network. Traces can be captured in real-time or off-line for less impact on the network and stored locally to flash, TFTP, FTP or in real time to a TZSP host running OmniPeek or Wireshark. For details on utilizing WiNG 5 capture features, please refer to the feature guide at: http://compass.motsolutions.com/doc/375558309/how_to_wing5_pktcap_v1.4_final.pdf 19
WiNG 5 Trace Collection Overview WiNG 5 provides several physical and logical points at which trace collection can take place. VPN Router The diagram is representative of a WiNG 5 access point and shows the many local interfaces from which captures can be collected. Bridge VLANs Bridge Additionally, the remotedebug feature of WiNG 5 allows for remote capture at a specified device, like a distributed sniffer. Ethernet Interface WLAN s Radio 20
Section III: Trace Collection Procedures 21
You should synchronize the clocks of all capture PCs to the correct time! 22
Trace Collection Considerations Understanding relative time of a trace and occurring problems is important when troubleshooting. It is good practice to sync the time on all capture machines and to reflect the time in the capture In OmniPeek, click once on the column headings to bring up the Packet List Options dialog, then select Absolute Time. 23
Trace Collection Considerations In Wireshark, right-click on the column bar, and select Column preferences in the menu. This will bring up the Wireshark preferences dialog. 24
Trace Collection Considerations You can high-light the default Time column and then change it to Absolute Time from the drop-down. 25
Collection Considerations: Ethernet Some adapters will strip vlan tags by default, when processing traffic. In order to make sure this information is included in your trace, ensure the driver allows for and VLAN processing is enabled. 26
Wireless Capture Placement When troubleshooting an AP, capture as close to the target AP as possible. In WiNG 5 this can be done at the target AP or a neighboring AP. When troubleshooting MU(s), place the capture device as close to the MU as possible or capture at the AP the MU is trying to associate to. MU Wireless capture Troubleshooting the AP Wireless Capture Troubleshooting the MU 27
Wireless Channel Considerations When collecting traces, many utilities will scan and capture on all available channels. This may cause some packets to be missed. It is best to lock onto a channel, matching that of an AP clients are trying to associate to. However, there are times when capturing on multiple channels is necessary in order to get traffic from all MU s in an area. With WiNG 5 s remote-debug command, this can easily be accomplished. When capturing from multiple hosts using remote-debug, the device at which the command is performed (typically a controller) will automatically collate the captures from multiple devices (AP s) into one stream for analysis. The following example initiates a capture at two access points on radio 1 for each. These may represent two AP s in a specific area, on different channels: remote-debug live-pktcap hosts ap7131-970408 ap7131-9313cc radio 1 28
Wired Capture Placement Traffic should be captured as close as possible to all devices related to the specific data conversation. This can be accomplished using switch span ports and / or capturing on WiNG 5 device interfaces Wired Capture PC on span port Server with issues WiNG 5 RF Switch Server with no issues WiNG 5 AP s Wired Capture PC on span port 29
Principles for Trace Collection A trace is only as good as the context in which is was captured. Give the trace a descriptive name Include date, customer name, MAC addr (if possible), SPR #. If wired trace, include location (srvr / ap / etc.), trunk # Example: 0506RamaSPR11008Ch6.apc 0710BellCanadaAPreset.pkt Should trace be L2 or L3? Do not use capture filters; filtering can be performed later. If possible, capture in continuous mode Problems may take time to manifest 30
Troubleshooting Tips Required information for debugging: Syslog Messages Syslog server connected to problem LAN Wired trace of all traffic into and out of the RF switch, in line with suspect traffic. Wireless traces taken at AP(s), or as near as possible to problem clients / AP s. Time of failure, assuming time synchronization Mac addresses of failed clients, servers, etc. Network topology diagram Narrative of problem, how and where the trace(s) was taken. 31
Trace Collection Wireless Issue Definition: Problem only occurs with MU / client. Examples Include Wireless Association failure Roaming issue Proxy ARP Traces to collect Wireless trace at client or AP Collection laptop at location of client Collection at AP radio interface or wireless interface Wired trace on segment client is on Spanned switch port Ge1 interface of AP using pktcap or remote-debug commands 32
Trace Collection Wireless Issues Wired Capture PC on span port Server with issues Capture here Wireless Capture PC w/ compatible adapter WiNG 5 RF Switch WiNG 5 AP s Server with no issues Wireless Client device (MU) Or Capture here on radio or wireless interface (using pktcap WiNG 5 command). 33
Trace Collection Firewall / Routing Issue Definition: Involves two endpoint in separate IP domains. Examples Include Can t access Internet (LAN WAN) VPN not working (LAN WAN) Outside can t access internal server (WAN LAN) Traces to collect Wired traces on each IP segment Span port on each subnet to capture traffic from wireless AP (client traversal) and wired destination (server / voice gateway, etc.) Wireless trace not needed 34
WiNG 5 Command Summary Simple capture to flash memory of WiNG 5 device: rfs4000-22d26e#service pktcap on interface ge1 write pktcaptest.pcap Capture / send to TZSP host for real-time analysis: TZSP host is running Wireshark and iperf.exe in server mode* rfs4000-22d26e#remote-debug live-pktcap hosts ap7131-970408 ap7131-9313cc write tzsp 192.168.150.10 Tazmen Sniffer Protocol is an encapsulation protocol used to wrap other protocols; typically in UDP and is used for wireless captures. WiNG 5 implementation of TZSP sends on UDP port 37008. *Iperf.exe is a free Windows CLI tool used for performance testing. You can initiate iperf in server mode, listening on UDP port 37008, so you don t receive ICMP destination port unreachable messages in your trace. iperf.exe -s -u -p 37008 35
WiNG 5 Command Summary When using the remote-debug command to capture on multiple hosts, the independent captures will be collated into a single stream at the initiated device (usually a controller). Because the actual trace is distributed among multiple devices, there will not be a significant load the controller / initiating device. The exception to this is if a capture is done using the rf-domain option, which then captures on all hosts in the rf-domain. This may cause too many packets too quickly for the initiating device to collate without dropping some packets. 36
Basic Trace Analysis Everyone taking traces should, at the least, be able to look at the capture file and see if there is data to and from the MU and the host app. in the trace. Traces of one-way communication do not aid in determining the problem. Make sure entire conversation is captured. Traces that do not capture the failure taking place also are of no use. Make sure the failure takes place and is captured in your trace file. 37
Basic Trace Analysis Perform quick filtering by right-clicking on packet and choosing Select Related Packets Group by source / destination Mac address Group by source / destination IP address Group by protocol Traces to collect Wired traces on each IP segment Span port on each subnet to capture traffic from wireless AP (client traversal) and wired destination (server / voice gateway, etc.) Wireless trace not needed 38
OmniPeek Quick Filters 39
OmniPeek Quick Filters OmniPeek will highlight all packets related to your selection and then you can choose what to hide. Filtering can always be done later, so when performing a capture, capture everything. 40
Wireshark Quick Filters 41
Wireshark Quick Filters Wireshark has the ability to build quick filters based on specific parts of the packet headers. Simply right-click the data to filter and select Apply as or Prepare as filter. The display filter box will be instantly populated with the filter syntax. 42
Additional Resources WiNG 5 Packet Capture Feature Guide: http://compass.motsolutions.com/doc/375558309/how_to_wing5_pktcap_v1. 4_final.pdf Packet Capture Screencasts: http://compass.motsolutions.com/web/wlan/how%20to%20videos iperf.exe for Windows https://publishing.ucf.edu/sites/itr/cst/pages/iperf.aspx 43