SER1289BU vsphere Troubleshooting Tips and Tricks Javier Menendez & Francis Daly #VMworld #SER1289BU
Agenda 1 Use of the VMware vsphere Command-line Interface (vcli), the ESXi Shell, and Log Files to Diagnose and Correct Problems 2 Troubleshooting Networking Issues 3 Troubleshooting Storage Issues 4 Troubleshooting vsphere HA Cluster and Performance Issues VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 2
Scope This Presentation covers portions of the Troubleshooting Workshop: For more information, go to http://vmware.com/education Certain troubleshooting topics are addressed in the VMware vsphere: Fast Track [6.5] course. 3
Troubleshooting Overview
Troubleshooting Process The troubleshooting process consists of the following tasks: Defining the problem Identifying symptoms Gathering information Identifying the cause of the problem Identifying possible causes Determining the root cause Solving the problem Identifying possible solutions Implementing the best solution #SER1289BU CONFIDENTIAL 5
Solving the Problem Can I Do It with the Web Client or Should I Use the esxi Shell? What are my choices: Can I fix the problem with the Web, Host or vsphere Client? Do I want or need to use the command line utilities? #SER1289BU CONFIDENTIAL 6
Troubleshooting Tools and Choices
Command-line Troubleshooting Choices Choices to obtain command-line access on a VMware ESXi host: VMware vsphere ESXi Shell Can be access via: 1. The DCUI (if enabled via ALT + F1) 2. Via putty after enabling ssh. vsphere Management Assistant (VMA) Appliance deployed via ovf Contains vcli package Does NOT use port 22. #SER1289BU CONFIDENTIAL 8
The esxcli Command The esxcli command offers the following namespaces, as well as many new esxcli functions added in the vsphere 6.5. If working with vsan, check out the new esxcli vsan debug namespace #SER1289BU CONFIDENTIAL 9
Example: The esxcli vsan Namespace The vsan related commands are found in esxcli vsan #SER1289BU CONFIDENTIAL 10
New Commands Added with Every New vsphere Release. Ex: The esxcli vsan debug Namespace The new vsan debug related commands are amazing to discover the state of your objects, vms, etc. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 11
Tip: Remember the esxcli esxcli command list Command esxcli esxcli command list displays all the command options. To quickly find the command needed, combine it with the grep command VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 12
The VMA s vicfg-* Commands vicfg- [tab] [tab] Commands with the vicfg- prefix enable you to manage your storage, network, and host configuration. Type vicfg- followed by the TAB key twice so see all the available commands. For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds. #SER1289BU CONFIDENTIAL 13
The esxi shell s esxcfg- Commands Are Still There esxcfg- [tab] [tab] Commands with the esxcfgprefix enable you to manage your storage, network, and host configuration. Type esxcfg- followed by the TAB key twice so see all the available commands. Remember the h (help) option to master these commands. #SER1289BU CONFIDENTIAL 14
Tip: Remember the -h Option for esxcfg-* Commands Use the h option (for help) to see options for this particular command and in some cases, even examples on how to use the command. How many times did you struggle to create a vmkernel port with the cli? VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 15
Manipulating Virtual Machines with the cli from the V.M.A.? vmware-cmd The vmware-cmd command is used to interact with virtual machines. vmware-cmd The vmware-cmd command is exclusively used for virtual machines. #SER1289BU CONFIDENTIAL 16
Example: Listing Your Virtual Machines vmware-cmd -l Lists the virtual machines that are located on the target host. Lists virtual machines by path to the.vmx file. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 17
Tip: Learn More on Your Own Type vmware-cmd --help to see what the command allows you to do. vmware-cmd allows you to start a vm, take snapshots, register a vm, etc. VMworld 2017 Content: Not for The esxi host has a similar command to manipulate vms called vim-cmd vmsvc. publication #SER1289BU CONFIDENTIAL 18
Manipulating Virtual Machines from the esxi shell vim-cmd vmsvc can perform similar tasks from the esxi shell. Wondering what the destroy option does? Better try it on a test vm. #SER1289BU CONFIDENTIAL 19
Tip: Location of VMware vcenter Server 6.5 Log Files The VMware vcenter Server 6.5 logs are located in the %ALLUSERSPROFILE%\VM Ware\vCenterServer\logs folder. Recommendation The VMware vcenter Server Appliance 6.5 logs are located in the /var/log/vmware folder. See http://kb.vmware.com/kb/2110014 and 1021806 for additional details. #SER1289BU CONFIDENTIAL 20
Useful ESXi Host Logs for Troubleshooting ESXi hosts write to multiple log files, depending on which action is being performed. Log file hostd.log syslog.log vmkernel.log vmkwarning.log vmksummary.log Purpose Host Management service logs Management service initialization, watchdogs, scheduled tasks, and DCUI use Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startups A summary of warning and alert log messages excerpted from the VMKernel logs A summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption #SER1289BU CONFIDENTIAL 21
Tip: Other Critical Logs to Be Aware of and How to View Them /var/log/fdm.log displays HA cluster related events and notifications. /var/log/shell.log displays commands executed inside of the esxi shell. VMworld 2017 Content: Not for /var/log/clomd.log displays vsan health related information for devices among the output publication #SER1289BU CONFIDENTIAL 22
Tip: Remember /var/log Lots of logs available in this directory #SER1289BU CONFIDENTIAL 23
Collecting Diagnostic Data for VMware Technical Support Methods for collecting diagnostic information to send to VMware Technical Support include the following: Use the GUI to export files to a log bundle. vsphere Client or vsphere Web Client Use the vm-support command to collect information from an individual ESXi host. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 24
Tip: Running vm-support when Engaging Support #SER1289BU CONFIDENTIAL 25
Network Troubleshooting Commands VMworld 2017 Content: Not for publication
Tip: Useful Networking Related Commands Use esxcfg-vmknic l or vicfg-vmknic -l to view your vmkernel port related information. Are they still there and enabled? Use esxcfg-nics l or vicfgnics l to view uplink related information. Are the nics up or down? Use esxcfg-vswitch l or vicfg-vswitch l to view virtual switch related information. Is the switch connected to at least one uplink? esxcli also has options to perform these tasks. For example: esxcli network nic list is similar to esxcfg-nics -l #SER1289BU CONFIDENTIAL 27
Tip: Did You Know That Somebody could accidentally down a nic? If so, type esxcli network nic up n vmnic2. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 28
Tip: Did You Know That Somebody could disable a vmkernel port with commands like esxcfg-vmknic -D? VMworld 2017 Content: Not for If so, type esxcfg-vmknic -e to reenable the vmkernel port. publication #SER1289BU CONFIDENTIAL 29
Tip: Rebuilding the Management Network from Scratch Somebody could accidentally disconnect an uplink from a virtual switch? ffaaa #SER1289BU CONFIDENTIAL 30
Need a Packet Sniffer? Remember pktcap-uw There is a built-in packet sniffer called pktcap-uw and an older one called tcpdump-uw. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 31
Storage Troubleshooting
Tip: Did You Know That You can see how many dastastores exist and how full the datastores are? This command displays vmfs, nfs, vsan and vvol datastores. There is also a vdf h command. #SER1289BU CONFIDENTIAL 33
Tip: Did You Know That You can easily view all sorts of information with esxcli? #SER1289BU CONFIDENTIAL 34
Tip: Use esxcli Storage Core Device Smart for SSD Information vsphere 6.5 Includes Use this command to retrieve data about a specific SSD device. Critical for vsan, Flash Read Cache or any datastore using SSDs. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 35 35
Tip: Be Familiar with partedutil partedutil is the command that partitions luns. It can also create, resize and delete partitions. #SER1289BU CONFIDENTIAL 36
Tip: Suspect Metadata Corruption? voma is the on-disk metadata analyzer for vmfs datastores. #SER1289BU CONFIDENTIAL 37
Troubleshooting Clusters and Performance VMworld 2017 Content: Not for publication
vsphere HA A reliable network connection between the hosts and VMware vcenter Server is essential for enabling vsphere HA. FDM ESXi host (slave) vcenter Server FDM vpxa hostd vpxa hostd vpxa ESXi host (slave) vpxd FDM hostd ESXi host (master) Heartbeat Datastores Management Network #SER1289BU CONFIDENTIAL 39
Tip: Remember F.D.M There is a log for HA that you can t see with the Web or vsphere Client? #SER1289BU CONFIDENTIAL 40
Tip: Did You Know That You can stop, start or restart the HA service from the cli if needed? #SER1289BU CONFIDENTIAL 41
Tip: You Can Restart All Services at Once If Needed Remember services.sh restart #SER1289BU CONFIDENTIAL 42
Tip: Did You Know That esxtop is an amazing tool for performance troubleshooting? VMworld 2017 esxtop has options for CPU (c), RAM (m), Disk (d,u,v) and Network (n) related statistics? esxtop has an option for vsan (-x option)? Content: Not for publication #SER1289BU CONFIDENTIAL 43
Tip: Did You Know That esxtop has built-in help? #SER1289BU CONFIDENTIAL 44
Tip: Did You Know That You can add more output to esxtop by pressing f? For example, you can add NUMA related information (G Option) VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 45
Tip: Learn the vsish Command and Its Capabilities vsish has hundreds of subcommands. Just type the command and take a look You can type commands like cd to change directories and get to view information. VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 46
Tip: Learn the vsish Command and Its Capabilities vsish has hundreds of subcommands. Just type the command and take a look VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 47
Global Support Services Learn more about how VMware is radically transforming Customer Support through VMware Skyline technology. See a demo in the VMware booth in the Solutions Exchange Sign up for a Meet the Experts roundtable in the Schedule Builder on the VMworld mobile app or visit the Meet the Experts, Level 2 Visit www.vmware.com/support/service/skyline VMworld 2017 Content: Not for publication #SER1289BU CONFIDENTIAL 48
Additional Education Resources At VMworld 2017 Education & Certification Lounge: VM Village Certification Exam Center: Jasmine EFG, Level 3 Online vsphere Training: www.vmware.com/go/vsphere65training VMware Training: www.vmware.com/education VMware Certification: www.vmware.com/certification or Save 50% distribution off VCP & VCAP exams at VMworld 2017 #SER1289BU CONFIDENTIAL 49