Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Size: px

Start display at page:

Download "Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme"

Randall Nash
6 years ago
Views:

1 SER2965BU Advanced Troubleshooting of ESXi Server 6.x for vsphere Gurus VMworld 2017 Content: Not for publication Ragavendra P Kumar & Abhilash Kunhappan #VMworld #SER2965BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #SER2965BU CONFIDENTIAL 2

3 #SER2965BU CONFIDENTIAL 3

4 chmod 777 <user> #SER2965BU CONFIDENTIAL 4

5 Do You Know? 2. Commands? in ESXi Server 1. Configuration Files? 3. Log Files? #SER2965BU CONFIDENTIAL 5

6 Learner Objectives Improvise troubleshooting steps for a vsphere admin by Understanding the ESXi Architecture from a component breakdown perspective Learn 7 important and key :- 1. Log Files : which log files to review and when 2. ESXi Commands : to isolate and troubleshoot issues 3. Configuration Files : forms the backbone of troubleshooting sessions Every VMworld session is time bound and we will cover as much as we can. For all remaining questions, we are always a phone call away. #SER2965BU CONFIDENTIAL 6

7 Agenda 1 7 Important Log files 2 7 Key ESXi Commands 3 7 Significant Configuration Files #SER2965BU CONFIDENTIAL 7

ESXi Architecture HTML 5 - vsphere WebClient HTML 5 ESXi Host Client Hostd esxcli localcli Shell VIM Clients Power CLI vpxa FDM VMX Process User worlds vnic DVFilter Sys Calls VIM vpxd HA DRS FT

8 ESXi Architecture HTML 5 - vsphere WebClient HTML 5 ESXi Host Client Hostd esxcli localcli Shell VIM Clients Power CLI vpxa FDM VMX Process User worlds vnic DVFilter Sys Calls VIM vpxd HA DRS FT vcenter Server User world Support Resource Scheduler Network Stack Storage Stack vmkapi and Interrupt handler vscsi VMkernel VMK Call Guest OS & Apps vcpu VMM Monitor Server Hardware #SER2965BU CONFIDENTIAL 8

9 ESXi Log Files #SER2965BU CONFIDENTIAL 9

10 1. vmksummary.log Abrupt Host reboots Time stamp indicating ESXi server reboot VMkernel will always track Host reboots, the below shows host rebooting has not been tracked Disable the Automated Server Recovery feature from the BIOS to avoid automatic reboots #SER2965BU CONFIDENTIAL 10

11 2. boot.gz log - Slow Boot Issues To troubleshoot both slow boot and boot process hang of ESXi server Location : /var/log/boot.gz Use DCUI - Alt + F12 to review VMkernel messages during this time Enabling Serial logging using (Shift + o) at the start up screen for log redirection vmkbootverbose=true vmblog=true logport=com1 debuglogtoserial=1 logport=com1 no-auto-partition bootuuid=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx #SER2965BU CONFIDENTIAL 11

12 2. boot.gz log - Slow Boot Issues Slow ESXi Boot Issues when, RDM devices are not perennially tagged Validate the esx.conf file with a key word search perennialresvd with respect to the NAA.ID of the LUN Command to set the perennially reserve tag per LUN # esxcli storage core device setconfig -d naa.id --perennially-reserved=true #SER2965BU CONFIDENTIAL 12

13 3. hostd & hostd-probe.log ESXi Not Responding hostd unresponsive and Host Not responding from vcenter server A successful hostd process start up or working fine hostd process should finish reading and populating the vminventory.xml file or Direct Connect to the ESXi with Host client should also validate the hostd status distribution Do NOT restart the ESXi Server when hostd is in a loop if you need to pursue the cause #SER2965BU CONFIDENTIAL 13

14 3. hostd.log ESXi Not Responding Understanding the Hostd status and validating the Hostd service restart is successful Key words search in hostd.log BEGIN SERVICES #SER2965BU CONFIDENTIAL 14

15 4. vmware.log - Virtual Machine Issues VM Power ON issues Attempt a Virtual Machine power ON and then validate the below log Identify the cause of the Power ON failure #SER2965BU CONFIDENTIAL 15

16 5. vmkernel.log - Storage Issues Understanding SCSI Sense codes are key for Storage issues SCSI codes which are T10 org defined codes SCSI sense codes #SER2965BU CONFIDENTIAL 16

17 6. vobd.log - Network & Storage Issues Network showing uplink transition Storage disconnect messages #SER2965BU CONFIDENTIAL 17

18 7. fdm.log High Availability (HA) Issues to identify :- the HostID? the Host is a Slave or Master? the Host that was isolated from cluster? #SER2965BU CONFIDENTIAL 18

19 ESXi Commands #SER2965BU CONFIDENTIAL 19

20 1. To Monitor & Configure ESXi Server Settings nic firewall vswitch san vvol vmfs boot module process 03 VMworld 2017 Content: Not for esxcli 04 vib profile sources cluster network datastore publication list kill process #SER2965BU CONFIDENTIAL 20

21 # esxcli VMX World ID information VMX Cartel ID information Displaying the ARP and Neighbor Discovery cache #SER2965BU CONFIDENTIAL 21

22 2. VMkernel Sysinfo Shell Command ** # vsish get <opt.1>/<opt.2>/<opt.3>,. hardware /bios /hardwareinfo net /pnics/vmnic<>/stats /portsets/vswitch<>/stats storage or distribution /scsifw/devices/.<>/stats /scsifw/adapters/.<>/stats ** take assistance always from GSS when in doubt #SER2965BU CONFIDENTIAL 22

23 # vsish VMworld 2017 Get the GroupID Get the VMX Cartel ID Content: Not for publication #SER2965BU CONFIDENTIAL 23

24 3. To Manage ESXi & VM Configurations net storage firmware getallvms power.getstate get.guest login connect registervm 03 vim-cmd 04 auth task_list task_cancel port_info service_list remove_service vprobes shutdown refresh_pnic #SER2965BU CONFIDENTIAL 24

25 # vim-cmd Validates if the Hostd is running by listing the VM s registered VMworld 2017 Content: Not for List the tasks running on the ESXi host publication #SER2965BU CONFIDENTIAL 25

26 4. To Manage VMFS Volumes & Virtual Disks # vmkfstools <opt.1> name.vmdk <opt.2> -e -v10 to check VM Snapshot Chain consistency like PID & CID in vmdks -t10 -v10 VMworld 2017 Content: Not for to check Inode integrity of a vmdk file in VMFS layer and not in GOS level publication -x. check, repair to check & correct basic file level issues of a vmdk file in VMFS layer #SER2965BU CONFIDENTIAL 26

27 # vmkfstools Check or Repair the Virtual Disk VMworld 2017 Content: Not for VM Snapshot Chain Consistency check publication #SER2965BU CONFIDENTIAL 27

28 5. # memstats to Get Detailed Memory Statistics VM is identified by the VMX cartel ID using the command ps grep i <vmx name> Memstats r vm-stats s name:<choose the options needed> VMworld 2017 Get the VMX cartel ID Virtual Machine Stats check based on the VMX cartel ID Content: Not for publication #SER2965BU CONFIDENTIAL 28

29 # memstats VM is identified by the VMX cartel ID using the command ps grep i <vmx name> Memstats r swap-stats s name:<choose the options needed> -u mb VMworld 2017 Content: Not for Swap Statistics on the Virtual Machine publication #SER2965BU CONFIDENTIAL 29

6. # pktcap-uw - Network Packet Capture Analysis Tool pktcap-uw --trace -- vmk <vmk port> pktcap-uw --trace -- uplink <uplink name> VMworld 2017 Output can be redirected to a file as well

30 6. # pktcap-uw - Network Packet Capture Analysis Tool pktcap-uw --trace -- vmk <vmk port> pktcap-uw --trace -- uplink <uplink name> VMworld 2017 Output can be redirected to a file as well Shows packet traverse path from each layer and to identify drop within the ESXi host MAC address of the Port that is being captured Content: Not for publication #SER2965BU CONFIDENTIAL 30

31 7. To Monitor Core ESXi Performance Power (usage & management) network (vswitch, uplinks) vsan n p x storage (LUN specific) v esxtop u virtual machine c d m CPU memory disk (hba level info) #SER2965BU CONFIDENTIAL 31

32 # esxtop #SER2965BU CONFIDENTIAL 32

33 ESXi 7 Effective Commands for Troubleshooting 1. esxcli : Monitoring or configuring ESXi infra settings 2. vim-cmd : Managing ESXi server & Virtual Machine related configurations 3. vsish : VMkernel Sys Info Shell 4. vmkfstools : for management of VMFS volumes & virtual disks from command line 5. memstats : Memory statistics 6. pktcap-uw : An enhanced Network packet capture analysis tool 7. esxtop : Real time performance monitoring #SER2965BU CONFIDENTIAL 33

34 ESXi Configuration Files #SER2965BU CONFIDENTIAL 34

35 7 Configuration File 1. /etc/vmware/esx.conf : Holds information about Storage, Network and Hardware 2. /etc/vmware/hostd/vminventory.xml : Virtual Machine Inventory list update on the ESXi 3. /etc/vmware/hostd/authorization.xml : vcenter to ESXi host connection authorization 4. /etc/vmware/vpxa/vpxa.cfg : vcenter and ESXi connectivity 5. /etc/vmware/vmkiscsid/iscsid.conf : iscsi configuration file 6. /etc/vmware/fdm : FDM config file with Clusterconfig, Hostlist and VM metadata 7. /etc/vmware/license.cfg : License configuration file for ESXi host #SER2965BU CONFIDENTIAL 35

36 Q & A #SER2965BU CONFIDENTIAL 36

Agenda 1 Use of the VMware vsphere Command-line Interface (vcli), the ESXi Shell, and Log Files to Diagnose and Correct Problems 2 Troubleshooting Net

SER1289BU vsphere Troubleshooting Tips and Tricks Javier Menendez & Francis Daly #VMworld #SER1289BU Agenda 1 Use of the VMware vsphere Command-line Interface (vcli), the ESXi Shell, and Log Files to Diagnose