Table of Contents HOL CHG

Transcription

1 Table of Contents Lab Overview - - vsphere Challenge Lab... 2 Lab Guidance... 3 Module 1 - Let's Start with the Basics (30 minutes)... 8 Introduction... 9 Can't vmotion a VM Can't ping a VM Module 2 - Basic Operations Management (30 minutes) Introduction Update VMware Tools My PowerCLI script isn't working Module 3 - Basic Resource Management (30 minutes) Introduction Troubleshoot a Poorly Performing VM Troubleshoot a Poorly Performing Host Module 4 - Advanced Operations Management (45 minutes) Introduction Assigning the Correct Permissions My VM Won't Power On My Host is Out of Compliance Module 5 - Advanced Resource Management (45 minutes) Introduction My Application is Sensitive to Latency I Created a Resource Pool and now Performance is Worse vnuma Troubleshooting Module Hints Module 1 - Let's Start with the Basics Module 2 - Basic Operations Management Module 3 - Basic Resource Management Module 4 - Advanced Operations Management Module 5 - Advanced Resource Management Page 1

2 Lab Overview - - vsphere Challenge Lab Page 2

3 Lab Guidance Note: It will take more than 90 minutes to complete this lab. You should expect to only finish 3-4 of the modules during your time. The modules are independent of each other so you can start at the beginning of any module and proceed from there. You can use the Table of Contents to access any module of your choosing. The Table of Contents can be accessed in the upper right-hand corner of the Lab Manual. The vsphere 6 Challenge Lab will do exactly that, challenge you to solve common vsphere problems that many experience on a daily basis! Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. Will you be able to fix a PowerCLI script that is not working? Can you power on a VM that won't power on? It's time to put your vsphere skills to the test. Lab Module List: Module 1 - Let's Start with the Basics (30 minutes). Module 2 - Basic Operations Management (30 minutes). Module 3 - Basic Resource Management (30 minutes). Module 4 - Advanced Operations Management (45 minutes). Module 5 - Advanced Resource Management (45 minutes). Lab Captains: Nick Fritsch, Senior Technical Account Manager, USA This lab manual can be downloaded from the Hands-on Labs Document site found here: This lab may be available in other languages. To set your language preference and have a localized manual deployed with your lab, you may utilize this document to help guide you through the process: VMware Technology Network (VMTN) For additional hints and to discuss the challenges presented in the lab further, be sure to visit the VMware Technology Netowork (VMTN) Community Pages: Page 3

4 Location of the Main Console 1. The area in the RED box contains the Main Console. The Lab Manual is on the tab to the Right of the Main Console. 2. A particular lab may have additional consoles found on separate tabs in the upper left. You will be directed to open another specific console if needed. 3. Your lab starts with 90 minutes on the timer. The lab can not be saved. All your work must be done during the lab session. But you can click the EXTEND to increase your time. If you are at a VMware event, you can extend your lab time twice, for up to 30 minutes. Each click gives you an additional 15 minutes. Outside of VMware events, you can extend your lab time up to 9 hours and 30 minutes. Each click gives you an additional hour. Alternate Methods of Keyboard Data Entry During this module, you will input text into the Main Console. Besides directly typing it in, there are two very helpful methods of entering data which make it easier to enter complex data. Page 4

5 Click and Drag Lab Manual Content Into Console Active Window You can also click and drag text and Command Line Interface (CLI) commands directly from the Lab Manual into the active window in the Main Console. Accessing the Online International Keyboard You can also use the Online International Keyboard found in the Main Console. 1. Click on the Keyboard Icon found on the Windows Quick Launch Task Bar. Page 5

6 Click once in active console window In this example, you will use the Online Keyboard to enter the sign used in addresses. The sign is Shift-2 on US keyboard layouts. 1. Click once in the active console window. 2. Click on the Shift key. Click on key 1. Click on the "@ key". Notice sign entered in the active console window. Page 6

7 Activation Prompt or Watermark When you first start your lab, you may notice a watermark on the desktop indicating that Windows is not activated. One of the major benefits of virtualization is that virtual machines can be moved and run on any platform. The Hands-on Labs utilizes this benefit and we are able to run the labs out of multiple datacenters. However, these datacenters may not have identical processors, which triggers a Microsoft activation check through the Internet. Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoft licensing requirements. The lab that you are using is a self-contained pod and does not have full access to the Internet, which is required for Windows to verify the activation. Without full access to the Internet, this automated process fails and you see this watermark. This cosmetic issue has no effect on your lab. Look at the lower right portion of the screen Please check to see that your lab is finished all the startup routines and is ready for you to start. If you see anything other than "Ready", please wait a few minutes. If after 5 minutes your lab has not changed to "Ready", please ask for assistance. Page 7

8 Module 1 - Let's Start with the Basics (30 minutes) Page 8

9 Introduction We'll start with the basics within this module to get your troubleshooting engine warmed-up. In this lab, you'll be challenged to: Resolve an issue with a virtual machine that cannot be migrated to a different ESXi host (vmotion) Determine why a virtual machine cannot be contacted by ping, and resolve the issue. vsphere Client vs vsphere Flash Client In the vsphere Challenge Lab, you will use both the existing vsphere Web Client (based on Flash) and the new vsphere Client (based on HTML5). HTML5 is the future direction of the vsphere Client, so you will use it predominantly throughout this lab. If you feel more comfortable using the vsphere Web Client, a shortcut to it (labeled vsphere Flash Client) is bookmarked within Google Chrome. 1. The default client for the lab is the vsphere Client (based on HTML5). Use the Home button within Google Chrome to access the vsphere Client. 2. A bookmark to the vsphere Flash Client is available within Google Chrome. Note: As mentioned previously, the default client for this lab will be the vsphere Client based on HTML5. Due to the fact that the vsphere Client does not have full feature parity with the vsphere Web Client (based on Flash), there will be some sections where you will be directed to use the vsphere Web Client. Page 9

10 Module Switcher To prepare Module 1 for the challenges that will be presented to you, we need to run the Challenge Lab MS (Module Switcher). 1. Double-click on the Challenge Lab MS shortcut located on the Main Console desktop. 2. Select Start under Module 1. Wait for the PowerCLI script to finish before proceeding. Once the script has completed, press Enter to close the PowerCLI window. Page 10

11 Can't vmotion a VM With vmotion, you can change the host on which a virtual machine is running, or you can change both the host and the datastore of the virtual machine. When you migrate virtual machines with vmotion and choose to change only the host, the entire state of the virtual machine is moved to the new host. The associated virtual disk remains in the same location on storage that is shared between the two hosts. When you choose to change both the host and the datastore, the virtual machine state is moved to a new host and the virtual disk is moved to another datastore. vmotion migration to another host and datastore is possible in vsphere environments without shared storage. After the virtual machine state is migrated to the alternate host, the virtual machine runs on the new host. Migrations with vmotion are completely transparent to the running virtual machine. The state information includes the current memory content and all the information that defines and identifies the virtual machine. The memory content includes transaction data and the bits of the operating system and applications that are in the memory. The defining and identification information stored in the state includes all the data that maps to the virtual machine hardware elements, such as BIOS, devices, CPU, MAC addresses for the Ethernet cards, chip set states, registers, and so forth. When you migrate a virtual machine with vmotion, the new host for the virtual machine must meet compatibility requirements so that the migration can proceed. Migration with vmotion occurs in three stages: 1. When the migration with vmotion is requested, vcenter Server verifies that the existing virtual machine is in a stable state with its current host. 2. The virtual machine state information (memory, registers, and network connections) is copied to the target host. 3. The virtual machine resumes its activities on the new host. If errors occur during migration, the virtual machine reverts to its original state and location. Open Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. Page 11

12 If you did not Start Module 1 using the Challenge Lab Module Switcher outlined within the Introduction of this Module, please Stop and go back to the Introduction of Module 1 and follow the directions to Start the Module using the Challenge Lab Module Switcher. If you already completed this step, please proceed to the next page. Login to the vsphere Client 1. Select the check box "Use Windows session authentication". 2. Select Login. Find challenge-01a 1. Search for 'challenge-01a' using the Search box within the vsphere Client. 2. Select challenge-01a. Module 1, Challenge 1 Multiple Administrators have tried to migrate (vmotion) 'challenge-01a' to 'esx-04a', 'esx-05a' or 'esx-06a', as required by the Application Owner. Your challenge is to: Determine why challenge-01a is unable to be migrated to esx-04a, esx-05a or esx-06a, resolve the configuration issue and complete the migration as requested by the Application Owner. Need a Hint? Page 12

13 You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next step unless you want to see the solution to the challenge. Module 1, Challenge 1 Solution Reconfiguration of the vsphere environment increased the vcpu of 'challenge-01a' from 2 to 4. ESXi hosts 'esx-04a', 'esx-05a' and 'esx-06a' only have 2 CPU, therefore you cannot migrate 'challenge-01a' to any of these hosts. Therefore, in order to migrate 'challenge-01a' to 'esx-04a', 'esx-05a' or 'esx-06a', you need to modify the vcpu of 'challenge-01a' from 4 to 2. Let's take a look at how to reconfigure 'challenge-01a' for 2 vcpu so that we can complete the challenge. Edit challenge-01a Settings We need to modify the CPU settings of 'challenge-01a' to fit within the resource restrictions of 'esx-04a', 'esx-05a' or 'esx-06a'. 1. Select 'challenge-01a'. 2. Select Actions. 3. Select Edit Settings. Page 13

14 Modify CPU Settings 1. Select the CPU drop-down and change CPU from 4 to Select OK. Migrate challenge-01a 1. Right-click on 'challenge-01a'. 2. Select Migrate. Page 14

15 Change Compute Resource Only 1. Leave the default selection 'Change compute resource only'. 2. Select Next. Select Compute Resource 1. 'challenge-01a' needs to be migrated to 'esx-04a', 'esx-05a' or 'esx-06a'. Select one of these three ESXi hosts. 2. Select Next. The ESXi host 'esx-03a' is not an acceptable migration target for 'challenge-01a'. Page 15

16 Select Networks You should not need to make any changes on the Select networks window. 1. Select Next. Page 16

17 Ready to Complete Page 17

18 Can't ping a VM Before we proceed with the next challenge which involves troubleshooting a networking issue with a particular virtual machine, consider these best practices when you configure your network. To ensure a stable connection between vcenter Server, ESXi, and other products and services, do not set connection limits and timeouts between the products. Setting limits and timeouts can affect the packet flow and cause services interruption. Isolate from one another the networks for host management, vsphere vmotion, vsphere FT, and so on to improve security and performance. Dedicate a separate physical NIC to a group of virtual machines, or use Network I/ O Control and traffic shaping to guarantee bandwidth to the virtual machines. This separation also enables distributing a portion of the total networking workload across multiple CPUs. The isolated virtual machines can then better handle application traffic, for example, from a Web client. To physically separate network services and to dedicate a particular set of NICs to a specific network service, create a vsphere Standard Switch or vsphere Distributed Switch for each service. If this is not possible, separate network services on a single switch by attaching them to port groups with different VLAN IDs. In either case, verify with your network administrator that the networks or VLANs you choose are isolated from the rest of your environment and that no routers connect them. Keep the vsphere vmotion connection on a separate network. When migration with vmotion occurs, the contents of the guest operating systems memory is transmitted over the network. You can do this either by using VLANs to segment a single physical network or by using separate physical networks (the latter is preferable). For migration across IP subnets and for using separate pools of buffer and sockets, place traffic for vmotion on the vmotion TCP/IP stack, and traffic for migration of powered-off virtual machines and cloning on the Provisioning TCP/IP stack. You can add and remove network adapters from a standard or distributed switch without affecting the virtual machines or the network service that is running behind that switch. If you remove all the running hardware, the virtual machines can still communicate among themselves. If you leave one network adapter intact, all the virtual machines can still connect with the physical network. To protect your most sensitive virtual machines, deploy firewalls in virtual machines that route between virtual networks with uplinks to physical networks and pure virtual networks with no uplinks. For best performance, use VMXNET 3 virtual machine NICs. Physical network adapters connected to the same vsphere Standard Switch or vsphere Distributed Switch should also be connected to the same physical network. Page 18

19 Configure the same MTU on all VMkernel network adapters in a vsphere Distributed Switch. If several VMkernel network adapters, configured with different MTUs, are connected to vsphere distributed switches, you might experience network connectivity problems. Open a Command Prompt 1. Select the Command Prompt shortcut located on the taskbar on the Main Console. Ping challenge-02a 1. Enter the following text into the command prompt. ping challenge-02a -t The -t keeps the ping going infinitely which will allow you to test different solutions without having to repeat this step to determine if the problem is solved or not. Module 1, Challenge 2 Notice the ping response is 'Destination host unreachable'. Your challenge is to Determine why challenge-02a is unreachable by ping. Page 19

20 Need a Hint? You can also view hints on the VMware Technology Network Communities here: Do NOT proceed to the next page unless you want to view the solution to your challenge. Module 1, Challenge 2 Solution It appears that someone created a vsphere Standard Switch using the companies naming convention a long time ago. The company has standardized on the vsphere Distributed Switch however they forgot to delete this vsphere Standard Switch from 'esx-06a'. There are no network adapters attached to the vsphere Standard Switch that 'challenge-02a' is attached to, therefore it is unable to communicate on the network. Page 20

21 Edit Settings 1. Select challenge-02a within the Inventory Tree. 2. Select Actions. 3. Select Edit Settings. Page 21

22 Change Network Adapter 1. For Network adapter 1, select VM-RegionA01-vDS-COMP. 2. Select OK. Return to the command prompt. Is the 'challenge-02a' now responding to ping? If your continuous ping has stopped, restart the continuous ping using this command. ping challenge-02a -t Responding to Ping Page 22

23 Congratulations, 'challenge-02a' is now communicating on the network. Great job diagnosing and resolving the problem before your manager called you asking why they couldn't login to the server. Module Switcher Now that you have completed Module 1, we need to reset the lab for your next challenge. 1. Double-click on Challenge Lab MS (Module Switcher) located on the Main Console desktop. 2. Select Stop under Module 1. You've Finished Module 1 This concludes Module 1, Let's Start with the Basics. We hope you have enjoyed taking it. Please do not forget to fill out the survey when you are finished. If you have time reamining, you can continue to the next page to proceed to Module 2 or jump to another module that is part of this lab. Click on "More Options - Table of Contents" to quickly jump to a module within the manual. Module 1 - Let's Start with the Basics (30 minutes) Module 2 - Basic Operations Management (30 minutes) Module 3 - Basic Resource Management (30 minutes) Module 4 - Advanced Operations Management (45 minutes) Module 5 - Advanced Resource Management (45 minutes) Page 23

24 Module 2 - Basic Operations Management (30 minutes) Page 24

25 Introduction This module will challenge you on Basic Operations Management tasks. Create a shared VMware Tools respository then update VMware Tools on a Windows virtual machine. Resolve an issue with a PowerCLI script that continues to fail. vsphere Client vs vsphere Flash Client In the vsphere Challenge Lab, you will use both the existing vsphere Web Client (based on Flash) and the new vsphere Client (based on HTML5). HTML5 is the future direction of the vsphere Client, so you will use it predominantly throughout this lab. If you feel more comfortable using the vsphere Web Client, a shortcut to it (labeled vsphere Flash Client) is bookmarked within Google Chrome. 1. The default client for the lab is the vsphere Client (based on HTML5). Use the Home button within Google Chrome to access the vsphere Client. 2. A bookmark to the vsphere Flash Client is available within Google Chrome. Note: As mentioned previously, the default client for this lab will be the vsphere Client based on HTML5. Due to the fact that the vsphere Client does not have full feature parity with the vsphere Web Client (based on Flash), there will be some sections where you will be directed to use the vsphere Web Client. Page 25

26 Update VMware Tools VMware Tools is a set of services and modules that enable several features in VMware products for better management of, and seamless user interactions with, guests operating systems. For example, VMware Tools has the ability to: Pass messages from the host operating system to the guest operating system. Customize guest operating systems as a part of the vcenter Server and other VMware products. Run scripts that help automate guest operating system operations. The scripts run when the power state of the virtual machine changes. Synchronize the time in the guest operating system with the time on the host operating system. VMware Tools Lifecycle Management provides a simplified and scalable approach for installation and upgrade of VMware Tools. It includes a number of feature enhancements, driver-related enhancements, and support for new guest operating systems. Run the latest version of VMware Tools or use open-vm-tools distributed with the Linux OS distribution. Although a guest operating system can run without VMware Tools, always run the latest version of VMware Tools in your guest operating systems to access the latest features and updates. You can configure your virtual machine to automatically check for and apply VMware Tools upgrades each time you power on your virtual machines. For information about enabling automatic upgrade of VMware Tools on your virtual machines, see the vsphere Virtual Machine Administration Guide. More Information: VMware Tools User Guide VMware Tools Lifecycle: Why Tools Can Drive You Crazy productlocker When downloading ESXi from My VMware, you have the option to download ESXi with or without VMware Tools packaged within the ESXi installer. ESXi with VMware Tools included is ~300MB in size, half of which is made up of the VMware Tools source files. Therefore, you can save significant space within ESXi, especially when using features such as Auto Deploy, if using the ESXi installer that does not include VMware Tools. To help streamline the updating of VMware Tools in the future, you're first challenge within this module is to create a shared VMware Tools source folder on a datastore shared amongst all ESXi hosts and copy the contents of the Product Locker folder to that shared folder. Don't worry, we'll help guide you along the way. Page 26

27 Open Putty 1. From the Windows taskbar, select the Putty shorcut Module 2 does not require you to use the Challenge Lab Module Switcher to Start the Module. Please proceed to the next step. Connect to an ESXi Host All ESXi hosts within the vsphere environment are running vsphere 6.5, therefore we can connect to any of the ESXi hosts within the environment. 1. Select esx-01a. 2. Select Open. Page 27

28 Explore productlocker You will automatically connect to the ESXi host as root user. The following 3 commands will walk you through accessing the contents of the ProductLocker folder, which contain the VMware Tools installation source files. 1. cd /productlocker/ 2. cd vmtools 3. ls The productlocker folder contains the VMware Tools installation files for each type of Operating System. There are 3 types of files ISO - These files are mounted to the VM to install VMware Tools within the guest OS. SIG - These files are signature files for the ISO files to prevent tampering with or changing of the ISO files. MANIFEST - These files describe the names, versions and resources of the files/ drivers within VMware Tools. Page 28

29 Open Google Chrome Open Google Chrome from the Windows desktop. Or, open a new Tab if you still have Chrome open from a previous challenge. 1. Open Google Chrome using the shortcut on the taskbar. Login to vsphere Client 1. Select the check box "Use Windows session authentication". 2. Select Login. Access Storage Page 29

30 1. Select Menu. A dropdown will appear with a list of vcenter choices. 2. Select Storage. Module 2, Challenge 1 We've helped get you started down the right path, but now it's your turn to complete the challenge. Your challenge is to: Create a shared productlocker folder that can be accessed by all ESXi hosts. Copy the contents from the productlocker folder on an ESXi host to the shared productlocker folder. Tools you will possibly use to complete this challenge: vsphere Client WinSCP Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you are ready to see the solution to your challenge. Module 2, Challenge 1 Solution You will use WinSCP to copy the VMware Tools installation files to the Main Console. 1. Double-click on the 'WinSCP VMTools' shortcut on the desktop of the Main Console. The shortcut is configured to automatically take you to the 'vmtools' folder on an ESXi host in the cluster. You will be presented the following: Page 30

31 Page 31

32 Download VMware Tools Installation Files 1. Select all the files within the 'vmtools' folder on the right-hand side of WinSCP. By default, the left-hand side of WinSCP should default to the 'VMTools' folder located on the desktop of the Main Console (not shown). The 'VMTools' folder is currently empty. If not, browse to 'C:\Users\Administrator\Desktop\VMTools' on the left-hand side of WinSCP. 2. Download all of the files within the 'vmtools' folder (right-hand side of WinSCP) to the 'VMTools' folder (left-hand side of WinSCP). We need to copy the files we downloaded from the ESXi host to a folder shared amongst all ESXi hosts in the vsphere cluster. Page 32

33 Login to vsphere Client You should still be logged into the vsphere Client from previous steps. If not, login to the vsphere Client. 1. Select the check box "Use Windows session authentication" 2. Select Login. Access Storage 1. Select Menu. A dropdown will appear with a list of vcenter choices. 2. Select Storage. Page 33

34 Choose a Shared Datastore 1. Select RegionA01-ISCSI01-COMP01. Create a New Datastore Folder 1. Select Files. 2. Select New Folder. 3. Enter a folder name of Shared-VMTools. Page 34

35 Use WinSCP to Copy VMware Tools Installation Files to New Folder 1. Double-click on the WinSCP VMTools shortcut on the desktop of the Main Console. The shortcut is configured to automatically take you to the 'vmtools' folder on an ESXi host in the cluster. You need to locate the folder you created in the previous step on the VMFS datastore. Browse VMFS Page 35

36 Follow these steps within the folder tree on the right-side of WinSCP to reach the 'Shared-VMTools' folder you created in the 'RegionA01-ISCSI01-COMP01' datastore. 1. Select the drop-down arrow at the top of the WinSCP to move from the 'vmtools' folder to the 'vmfs' folder. Select the vmfs folder. 2. Select volumes. 3. Select RegionA01-ISCSI01-COMP01. Notice that the 'RegionA01-ISCSI01-COMP01' folder has a shortcut icon. This shortcut will take you to the original folder that has a 32-character alphanumeric name. This is important as upcoming steps will reference the 32-character alphanumeric, not the friendly name. For your information, the 32-character alphanumeric name is 5727fe90-1aaae6f fc3. 1. Select Shared-VMTools. Page 36

37 Copy VMware Tools Installation Files 1. Copy the VMware Tools installation files that you downloaded to the 'VMTools' folder (left-hand side of WinSCP) into the 'Shared-VMTools' folder (right-hands side of WinSCP) you created within the 'RegionA01-ISCSI01-COMP01' datastore. Page 37

38 Product Locker Location While not required as part of this challenge, there is an additional step needed to update each ESXi host with the new location of the VMware Tools installation files. Select a ESXi host and go to Configure > Advanced System Settings > UserVars.ProductLockerLocation. Edit the location to reflect the new productlocker folder you created: /vmfs/volumes/ RegionA01-ISCSI01-COMP01/Shared-VMTools If you have a large number of ESXi hosts, performing these steps on each ESXi host individually is not recommended. The following PowerCLI script provides you with a way to update all ESXi hosts at once. Connect to vcenter Server in PowerCLI Get-VMhost -Location <cluster name> Set-VMHostAdvancedConfiguration -Name UserVars.ProductLockerLocation -Value /vmfs/volumes/regiona01-iscsi01-comp01/productlocker A restart of ESXi is required to complete the updating of the productlocker location. This completes this challenge. Proceed to the next page to continue with Module 2 or use the Table of Contents to select a different challenge or module. Page 38

39 My PowerCLI script isn't working vsphere PowerCLI is a powerful command line tool that lets you automate all aspects of vsphere management, including networking, storage, virtual machines, tasks, and more. vsphere PowerCLI is distributed as a Windows PowerShell snap-in, and includes more than 300 PowerShell cmdlets, along with documentation and samples. More Information: VMware PowerCLI User's Guide VMware PowerCLI Cmdlet Reference Run the Get-VM-Snapshots PowerCLI script Page 39

40 Module 2, Challenge 2 You'll notice the 'Get-VM-Snapshots' PowerCLI script fails to run. Your challenge is to: Determine why the script failed to execute and resolve the issue. Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you want to view the solution to the challenge. Module 2, Challenge 2 Solution Did you figure it out? The error message will give you a hint to where the error is within the PowerCLI script. Focus on the characters after the path to Get-VM-Snapshot.ps1. 3 char:1 This indicates the issue with the PowerCLI script is located on line 3 of the script. Let's take a closer look at line 3 of the script. 1. Right click on Get-VM-Snapshots.ps1 within the Challenge Lab Scripts folder. 2. Select Edit with Notepad++. Page 40

41 The text on Line 3 begins with: Get-Snapshot The Get-Snapshot command is the incorrect command to start the script with. You'll notice that the Get-Snapshot command is used later within the script. The first command we need to run within the script is the 'Get-VM' command. This will retrieve a list of VMs within vcenter. After that, we run the 'Get-Snapshot' command to get a list of Snapshots for each VM. Therefore, change the 'Get-Snapshot' command on line 3 to 'Get-VM'. The PowerCLI script should look like the following: Connect-VIServer -Server 'vcsa-01a.corp.local' -User 'CORP\Administrator' -Password 'VMware1!' Get-VM Format-Table -VM $_ -VM $_ Measure-Object -Sum SizeMB).Sum}} Read-Host -Prompt "Press Enter to Exit" Once you've modified the script, Select Save then run the script again by right-clicking on the script and select Run with Powershell. Get VM Snapshot Success Page 41

42 After correcting the 'Get-VM-Snapshot.ps1' PowerCLI script and Runnig with Powershell, we can see the number and size of snapshots associated with each virtual machine. You've Finished Module 2 This concludes Module 2, Basic Operations Management. We hope you enjoyed taking it. Please do not forget to fill out the survey when you are finished. If you have time remaining, you can continue to the next page to begin Module 3, or you can jump to another Module within this lab. Click on "More Options - Table of Contents" to quickly jump to a challenge within the module. Module 1 - Let's Start with the Basics (30 minutes) Module 2 - Basic Operations Management (30 minutes) Module 3 - Basic Resource Management (30 minutes) Module 4 - Advanced Operations Management (45 minutes) Module 5 - Advanced Resource Management (45 minutes) Page 42

43 Module 3 - Basic Resource Management (30 minutes) Page 43

44 Introduction This module will challenge you on Basic Resource Management tasks. Troubleshoot a VM that has been performing poorly. Determine why an ESXi host continues to have degraded performance. vsphere Client vs vsphere Flash Client In the vsphere Challenge Lab, you will use both the existing vsphere Web Client (based on Flash) and the new vsphere Client (based on HTML5). HTML5 is the future direction of the vsphere Client, so you will use it predominantly throughout this lab. If you feel more comfortable using the vsphere Web Client, a shortcut to it (labeled vsphere Flash Client) is bookmarked within Google Chrome. 1. The default client for the lab is the vsphere Client (based on HTML5). Use the Home button within Google Chrome to access the vsphere Client. 2. A bookmark to the vsphere Flash Client is available within Google Chrome. Note: As mentioned previously, the default client for this lab will be the vsphere Client based on HTML5. Due to the fact that the vsphere Client does not have full feature parity with the vsphere Web Client (based on Flash), there will be some sections where you will be directed to use the vsphere Web Client. Page 44

45 Module Switcher To prepare Module 3 for the challenges that will be presented to you, we need to run the Challenge Lab MS (Module Switcher). 1. Double-click on the Challenge Lab MS shortcut located on the Main Console desktop. 2. Select Start under Module 3. Wait for the PowerCLI script to finish before proceeding. Once the script has completed, press Enter to close the PowerCLI window. Page 45

46 Troubleshoot a Poorly Performing VM Before we get started troubleshooting a poorly performing virtual machine, let's review some troubleshooting suggestions. The suggestions are not meant to be a comprehensive guide to diagnosing and troubleshooting problems in the virtual environment. It is meant to provide information about some common problems that can be solved without contacting VMware Technical Support. Temporary spikes in CPU usage indicate that you are making the best use of CPU resources. Consistently high CPU usage might indicate a problem. You can use the CPU performance charts to monitor CPU usage for hosts, clusters, resource pools, virtual machines, and vapps. Host machine memory is the hardware backing for guest virtual memory and guest physical memory. Host machine memory must be at least slightly larger than the combined active memory of the virtual machines on the host. A virtual machine's memory size must be slightly larger than the average guest memory usage. Increasing the virtual machine memory size results in more overhead memory usage. Datastores represent storage locations for virtual machine files. A storage location can be a VMFS volume, a directory on Network Attached Storage, or a local file system path. Datastores are platform-independent and hostindependent. Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example, you might notice a performance degradation with applications that frequently read from and write to the hard disk. If you see a spike in the number of disk read/write requests, check if any such applications were running at that time. Network performance is dependent on application workload and network configuration. Dropped network packets indicate a bottleneck in the network. Slow network performance can be a sign of load-balancing problems. More Information: vsphere Monitoring and Performance Monitoring Guest Operating System Performance Connect to challenge-01a Page 46

47 1. Connect to 'challenge-01a' by double-clicking on the 'chg-01a.rdp' shortuct on the Main Console desktop. Alternatively, you can right-click on 'chg-01a.rdp' and select Connect. If you did not Start Module 3 using the Challenge Lab Module Switcher outlined within the Introduction of this Module, please Stop and go back to the Introduction of Module 3 and follow the directions to Start the Module using the Challenge Lab Module Switcher. If you already completed this step, please proceed to the next page. Connect to challenge-04a Repeat the same steps for 'challenge-04a'. 1. Connect to 'challenge-04a' by double-clicking on the 'chg-04a.rdp' shortuct on the Main Console desktop. Alternatively, you can right-click on 'chg-04a.rdp' and select Connect. Wait to be logged into both 'challenge-01a' and 'challenge-04a' before proceeding to the next step. Start SPECjbb Both 'challenge-01a' and 'challenge-04a' has some tools installed on it to help you monitor its well-being. On the desktop, you can see information about the virtual machine including its hostname, IP address, CPU usage, and memory usage. The CPU usage and memory usage update in real time. In the CPU usage chart, you can see both the total CPU used ("Total:") and the percentage of CPU usage that is consumed by the guest operating system's kernel ("Krnl:"). 1. Double-click on SPECjbb to start the performance measurement application. Page 47

48 Start Benchmark A window titled SPECjbb2005 opens. A copy of SPECjbb2005 is installed on both 'challenge-01a' and 'challenge-04a'. SPECjbb2005 is a real SPEC java-based benchmark that can measure CPU and memory performance. 1. Select Start Benchmark to start the benchmark. Make sure you Start the Benchmark on both 'challenge-01a' and 'challenge-04a'. SPECjbb Starts Wait about a minute for the benchmark to begin running on both 'challenge-01a' and 'challenge-04a'. Page 48

49 Resize Windows Resize both the 'challenge-01a' and 'challenge-04a' desktops so that you can view the Performance results. Module 3, Challenge 1 As you can see, 'challenge-01a' is performing poorly when compared to 'challenge-04a'. Your challenge is to: Determine why challenge-01a is not performing similar to challenge-04a. Need a Hint? You can also view hints on the VMware Technology Network Communities here: Page 49

50 DO NOT proceed to the next page unless you are ready to view the Solution to this Challenge. Module 3, Challenge 1 Solution If you look closely, you'll notice CPU Shares have been setup for both 'challenge-01a' and 'challenge-04a'. The key difference though is 1000 Shares have been configured for 'challenge-01a' and 2000 Shares have been configured for 'challenge-04a'. This would explain why 'challenge-01a' is performing poorly when compared to 'challenge-04a'. Shares specify the relative importance of a virtual machine. In this instance, 'challenge-04a' had twice as many CPU Shares as 'challenge-01a' and therefore is entitled to consume twice as much of that resource when multiple virtual machines are competing for resources. There are a couple of ways to solve this challenge. 1. You can modify the CPU Shares for 'challenge-04a' to match that of 'challenge-01a' so that they both equal You can modify the CPU Shares for 'challenge-01a' to match that of 'challenge-04a' so that they both equal You can modify the CPU Shares for both 'challenge-01a' and 'challenge-04a' to both equal 0. We'll walk through modifying the CPU Shares to both equal 0 so that future VMs that reside on this host are not subjected to the same performance characteristics. Open Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. Page 50

51 Login to the vsphere Client 1. Select the check box "Use Windows session authentication". 2. Select Login. Edit Settings on challenge-01a 1. Find 'challenge-01a' within the Inventory Tree on the left hand side of the vsphere Client. Right click on 'challenge-01a' to launch the context menu. 2. Select Edit Settings. Page 51

52 Edit CPU Shares 1. Select the triangle next to CPU to expand the CPU menu. 2. Select the drop down menu next to where it currently states Custom and change the value to Normal. Repeat for challenge-04a Repeat the same process for 'challenge-04a'. 1. Find 'challenge-04a' within the tree on the left hand side of the vsphere Web Client. Right click on 'challenge-04a' to launch the context menu. 2. Select Edit Settings. 3. Select the triangle next to CPU to expand the CPU menu. 4. Select the drop down menu next to where it currently states Custom and change the value to Normal. If the SPECjbb Benchmark application is still running, the performance of 'challenge-01a' and 'challenge-04a' should equalize after a few cycles. If the SPECjbb Benchmark application is not running, feel free to start the Benchmark on each virtual machine again to ensure that the virtual machines are performing similarly. This completes this challenge. Proceed to the next page to continue with Module 3 or use the Table of Contents to select a different challenge or module. Page 52

53 Troubleshoot a Poorly Performing Host Before your next challenge, let's review some solutions for Consistently High CPU Usage. Temporary spikes in CPU usage indicate that you are making the best use of CPU resources. Consistently high CPU usage might indicate a problem. You can use the CPU performance charts to monitor CPU usage for hosts, clusters, resource pools, virtual machines, and vapps. Host CPU usage constantly is high. A high CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the host. Virtual machine CPU usage is above 90% and the CPU ready value is above 20%. Application performance is impacted. The host probably is lacking the CPU resources required to meet the demand. There might be too many virtual CPUs relative to the number of regular CPUs. There might be an IO storage or networking operation that places the CPU in a wait state. The Guest OS generates too much load for the CPU. Solutions Verify that VMware Tools is installed on every virtual machine on the host. Compare the CPU usage value of a virtual machine with the CPU usage of other virtual machines on the host or in the resource pool. The stacked bar chart on the host's Virtual Machine view shows the CPU usage for all virtual machines on the host. Determine whether the high ready time for the virtual machine resulted from its CPU usage time reaching the CPU limit setting. If so, increase the CPU limit on the virtual machine. Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the host might remain at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, set the CPU reservations for high-priority virtual machines to guarantee that they receive the required CPU cycles. Increase the amount of memory allocated to the virtual machine. This action decreases disk and or network activity for applications that cache. This might lower disk I/O and reduce the need for the host to virtualize the hardware. Virtual machines with smaller resource allocations generally accumulate more CPU ready time. Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. For example, a single-threaded application on a four-way virtual machine only benefits from a single vcpu. But the hypervisor's maintenance of the three idle vcpus takes CPU cycles that could be used for other work. Page 53

54 If the host is not already in a DRS cluster, add it to one. If the host is in a DRS cluster, increase the number of hosts and migrate one or more virtual machines onto the new host. Upgrade the physical CPUs or cores on the host if necessary. Use the newest version of hypervisor software, and enable CPU-saving features such as TCP Segmentation Offload, large memory pages, and jumbo frames. Now you may be thinking that this challenge is related Host CPU Performance. You may be right, but you may also be wrong. The only way to find out is to proceed with your challenge. Good luck! Launch Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. Login to vsphere Client 1. Select the check box "Use Windows session authentication". 2. Select Login. Browse to the Poorly Performing Host Page 54

55 The Operations team informed you that 'esx-01a' is performing poorly. We need to investigate further. 1. Select the host esx-01a. 2. Select Monitor. 3. Select Advanced. You may need to expand Performance to see Advanced by selecting the arrow to the left of Performance. The current virtual machine workload on 'esx-05a' is consuming 50% of the available CPU resources available. If you look closely at the Performance graph, you'll notice 5 different counters: green, blue, purple, orange and black. Using the Performance Chart Legend, we see that the blue counter represents CPU 0, the purple counter represents CPU 1, the green counter represents CPU 2, the orange counter represents CPU 3 and the black counter represents overall CPU usage for the host. View Memory Performance Modify the view to show Memory Usage. 'esx-01a' has 8GB of memory available to it. The current virtual machine workload is consuming close to 80% of available Memory. Investigate the performance of 'esx-01a' by using the Advanced Performance Monitor. Use the dropdown (CPU usage in %) in the upper right-hand corner of the Performance window to view metrics for Memory, Network and other host-based metrics. Module 3, Challenge 2 Your challenge is to: Alleviate the performance pressure on 'esx-01a'. Need a Hint? You can also view hints on the VMware Technology Network Communities here: Page 55

56 DO NOT proceed to the next page unless you are ready to view the Solution to this Challenge. Module 3, Challenge 2 Solution The ESXi host 'esx-01a' was performing fine up until the point when a colleague accidentally started the memhog (memory hog) virtual machine on the same host as 'challenge-02a'. The memhog virtual machine is only used to performance benchmarking within the environment. To solve the performance issue on 'esx-01a', you could've performed any of the following: Migrated either 'challenge-02a' or 'memhog' to a different ESXi host. Created a memory limit on 'challenge-02a' and/or 'memog'. Page 56

57 Stop Module 3 Before you celebrate with colleagues, open the Challenge Lab MS (module switcher) located on the desktop of the Main Console. 1. Double-click on the Challenge Lab MS (Module Switcher) shortcut located on the Main Console desktop. 2. Select Stop under Module 3. Wait for the PowerCLI window to close before proceeding to the next step. You've Finished Module 3 This concludes Module 3, Basic Resource Management. We hope you have enjoyed taking it. Please do not forget to fill out the survey when you are finished. If you have time remaining, you can proceed to the next page to begin Module 4 or you can jump to another Module within this lab. Click on "More Options - Table of Contents" to quickly jump to a challenge within the module. Module 1 - Let's Start with the Basics (30 minutes) Module 2 - Basic Operations Management (30 minutes) Module 3 - Basic Resource Management (30 minutes) Module 4 - Advanced Operations Management (45 minutes) Module 5 - Advanced Resource Management (45 minutes) Page 57

58 Module 4 - Advanced Operations Management (45 minutes) Page 58

59 Introduction This module will challenge you on Advanced Operations Management tasks. A new user needs specific permissions within the vsphere environment and it's up to you to make sure they are correct before this person makes an error, such as deleting every VM within the evironment. Determine why a VM will not Power ON. An ESXi host is Out of Compliance, how do I fix it? vsphere Client vs vsphere Flash Client In the vsphere Challenge Lab, you will use both the existing vsphere Web Client (based on Flash) and the new vsphere Client (based on HTML5). HTML5 is the future direction of the vsphere Client, so you will use it predominantly throughout this lab. If you feel more comfortable using the vsphere Web Client, a shortcut to it (labeled vsphere Flash Client) is bookmarked within Google Chrome. 1. The default client for the lab is the vsphere Client (based on HTML5). Use the Home button within Google Chrome to access the vsphere Client. 2. A bookmark to the vsphere Flash Client is available within Google Chrome. Note: As mentioned previously, the default client for this lab will be the vsphere Client based on HTML5. Due to the fact that the vsphere Client does not have full feature parity with the vsphere Web Client (based on Flash), there will be some sections where you will be directed to use the vsphere Web Client. Page 59

60 Module Switcher To prepare Module 4 for the challenges that will be presented to you, we need to run the Challenge Lab MS (Module Switcher). 1. Double-click on the Challenge Lab MS shortcut located on the Main Console desktop. 2. Select Start under Module 4. Wait for the PowerCLI script to finish before proceeding. Once the script has completed, press Enter to close the PowerCLI window. Page 60

61 Assigning the Correct Permissions Permissions are access roles that consist of a user and the users assigned role for an object such as a virtual machine or ESXi host. Permissions grant users the right to perform the activities specified by the role on the object to which the role is assigned. For example, to configure memory for the host, you must grant a role to a user that includes the Host > Configuration > Memory Configuration privilege. By assigning different roles to users for different objects, you control the tasks that users can perform in your vsphere environment. Users other than root and vpxuser initially have no permissions on any objects, which means they cannot view these objects or perform operations on them. A user with Administrator privileges must assign permissions to these users to allow them to perform tasks. The list of privileges is the same for ESXi and vcenter Server. Multiple Permissions Many tasks require permissions on more than one object. Permissions applied on a child object always override permissions that are applied on a parent object. Virtual machine folders and resource pools are equivalent levels in the hierarchy. If you assign propagating permissions to a user or group on a virtual machine's folder and its resource pool, the user has the privileges propagated from the resource pool and from the folder. If multiple group permissions are defined on the same object and the user belongs to two or more of those groups, two situations are possible: If no permission is defined for the user on that object, the user is assigned the set of privileges assigned to the groups for that object. If a permission is defined for the user on that object, the user's permission takes precedence over all group permissions. Permission Examples These rules can help you determine where you must assign permissions to allow particular operations: Any operation that consumes storage space, such as creating a virtual disk or taking a snapshot, requires the Datastore > Allocate Space privilege on the target datastore, as well as the privilege to perform the operation itself. Moving an object in the inventory hierarchy requires appropriate privileges on the object itself, the source parent object (such as a folder or cluster), and the destination parent object. Page 61

62 Each host and cluster has its own implicit resource pool that contains all the resources of that host or cluster. Deploying a virtual machine directly to a host or cluster requires the Resource > Assign Virtual Machine to Resource Pool privilege. Module 4, Challenge 1 If you did not Start Module 4 using the Challenge Lab Module Switcher outlined within the Introduction of this Module, please Stop and go back to the Introduction of Module 4 and follow the directions to Start the Module using the Challenge Lab Module Switcher. If you already completed this step, please proceed to the next page. The Developer Manager sent you an with the following permission requirements for the new Junior Developer. 1. Needs access to virtual machines and templates, including ability to provision new virtual machines from template. 2. Should not be able to add or remove CPU or memory from any virtual machines. 3. Complete snapshot capabilities including create, rename, remove and revert to for virtual machines. 4. Ability to create new folders. 5. Ability to perform all power operations for virtual machines. 6. Ability to clone virtual machines. An Active Directory account has already been created for the new developer within the CORP.LOCAL domain. Those details are as follows: Username: CORP\jdev Password: VMware1! Your challenge is to Assign the correct vcenter permissions to the Junior Developer based upon the requirements provided. Before you get started, your Director has implemented security standards, therefore all requested permissions for the Junior Developer should only be applied on the Development folder within vcenter. Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you want to view the solution to this challenge. Page 62

63 Module 4, Challenge 1 Solution Were you able to assign the correct permissions to the Junior Developer? Let's put your resolution to the test and walk through the solution. Open Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. Login to the vsphere Client 1. Select the check box "Use Windows session authentication". 2. Select Login. Page 63

64 Create New Role We need to create a new role with the requested permissions to assign to the Junior Developer. Create New Role 1. Select Roles. 2. Select the Plus sign to Add a new role. Page 64

65 Provisioning 1. Select Virtual machine. 2. Select Create template from virtual machine and Deploy template. Snapshot Management 1. Select Snapshot management. Selecting the 'Snapshot management' privilege will propogate all privilege's located within Snapshot management. Power Operations Page 65

66 Clone Virtual Machine 1. Select Clone virtual machine. That should cover all requested privileges for the Junior Developer. 2. Select Next. Name Role 1. Name the Role Junior Developer. 2. Select Finish. Page 66

67 Browse to the Development Folder 1. Select Menu. 2. Select VMs and Templates. Browse to Development Folder 1. Select the Development folder. You may need to expand 'vcsa-01a.corp.loca'l and 'RegionA01' in order to see the Development folder. 2. Select Permissions. 3. Select the Plus sign to add a new permission. Page 67

68 Add New Permission 1. Select CORP.LOCAL from the dropdown list. 2. Enter jdev. 3. Select the Junior Developer role from the dropdown list. 4. Check the box Propogate to children. 5. Select OK. Notice Development at the top of the window next to Add Permission. This helps you identify which vcenter object you are applying the new permission to. Permission Added This completes this challenge. Proceed to the next page to continue with Module 4 or use the Table of Contents to select a different challenge or module. Page 68

69 My VM Won't Power On You can often obtain valuable troubleshooting information by looking at the logs provided by the various services and agents that your implementation is using. Most logs are located in C:\ProgramData\VMware\vCenterServer\logs for Windows deployments or /var/log/ for vcenter Server Appliance deployments. Common logs are available in all implementations. Other logs are unique to certain deployment options (Management Node or Platform Services Controller). Common Logs applmgmt = VMware Appliance Management Service CloudVM = Logs for allotment and distribution of resources between services CM = VMware Component Manager FirstBoot = Location where first boot logs are stored rhttpproxy = Reverse Web Proxy SCA = VMware Service Control Agent statsmonitor = VMware Appliance Monitoring Server (VCSA only) Vapi = VMware vapi Endpoint vmaffd = VMware Authentication Framework daemon vmdird = VMware Directory Service daemon vmon = VMware Service Lifecycle Manager More Information: vsphere Troubleshooting Open Google Chrome 1. Launch Google Chrome using the shortcut on the Main Console desktop. Login to vsphere Client Page 69

70 1. Select the check box "Use Windows session authentication". 2. Select Login. Search for challenge-05a 1. Within the Search field located next to the Menu drop down, enter challenge-05a and press Enter. 2. Select challenge-05a. Power On challenge-05a 1. Select Actions. 2. Expand Power to see all available Power options. 3. Select Power On. Module 4, Challenge 2 1. Expand Recent Tasks by selecting it at the bottom of the window and check to make sure 'challenge-05a' powered on successfully. Page 70

71 It appears 'challenge-05a' won't power on. Examine the error message to begin troubleshooting the issue. Your challenge is to: Power on the challenge-05a virtual machine. Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you want to see the solution to the challenge. Module 4, Challenge 2 Solution Select the 'challenge-05a' virtual machine within the Inventory Tree. Notice that the virtual machine is configured for 8 CPU(s). This is preventing the 'challenge-05a' virtual machine from being powered-on as ESXi host 'esx-05a' does not have sufficient CPU Page 71

72 resources to meet the current CPU requirements (8 CPU) of the 'challenge-05a' virtual machine.. Edit Settings To resolve the misconfiguration issue for the 'challenge-05a' virtual machine: 1. Right-click on 'challenge-05a'. 2. Select Edit Settings. Correct CPU Configuration 1. Select the Core per Socket drop-down menu. Notice that the only options available to you are 1 or 2. Select 1 or 2. Page 72

73 2. Select OK. Notice that vcenter Server prevents you from misconfiguring the virtual machine within the User Interface. Now you may be asking yourself how the 'challenge-05a' virtual machine was configured with 8 CPU if the vcenter Server User Interface will not allow you to choose such a setting. The numvcpus line within the virtual machines VMX file (configuration file) was modified to 8 CPU. This shows the danger of modifying the VMX file directly, as it can override the measures put in-place within vcenter Server to prevent a misconfiguration. Challenge Completed Good job, after modifying the number of CPU from 8 to 1 or 2, the 'challenge-05a' virtual machine successfully powers on. This completes this challenge. Proceed to the next page to continue with Module 4 or use the Table of Contents to select a different challenge or module. Page 73

74 My Host is Out of Compliance The Host Profiles feature creates a profile that encapsulates the host configuration and helps to manage the host configuration, especially in environments where an administrator manages multiple hosts or clusters invcenter Server. Host Profiles provide an automated and centrally managed mechanism for host configuration andconfiguration compliance. Host Profiles can improve efficiency by reducing reliance upon repetitive, manualtasks. Host Profiles capture the configuration of a pre-configured and validated reference host, store the configuration as a managed object and use the catalog of parameters contained within to configure networking, storage, security, and other host-level parameters. Host Profiles can be applied to individual hosts, a cluster, or all the hosts and clusters associated to a host profile. Applying a Host Profile to a cluster affects all hosts in the cluster and result in a consistent configuration across the applied hosts. Host Profiles can be used to validate the configuration of a host by checking compliance of a host or cluster against the Host Profile that is associated with that host or cluster. More Information: vsphere Host Profiles Enhancements to Host Profiles in vsphere 6.5 Getting Started with Host Profiles Host Profiles is not available yet within the vsphere Client (HTML5), therefore we must use the current vsphere Flash Client. 1. Select the Google Chome shortcut on the taskbar. vsphere Flash Client 1. Select vsphere Flash Client from the Bookmarks Bar. Page 74

75 Login to vsphere Flash Client 1. Select the check box "Use Windows session authentication". 2. Select Login. Explore Host Profiles 1. Select the Home icon within vsphere Web Client. 2. Select Policies and Profiles. Page 75

76 Explore Host Profiles 1. Select Host Profiles from Navigator. Production Host Profile A team member has already created a Host Profile appropriately named 'Production Host Profile'. Notice that one host is currently attached to this host profile and it is showing as 'Not Compliant'. Let's determine why the host is showing as 'Not Compliant'. 1. Select Host Profile Compliance. Wait for the 'Check Compliance' task to complete. Use the Recent Tasks windows to check the status of the 'Check Compliance' task. Not Compliant Hosts After the Check Compliance task completes, notice that 1 ESXi host is not compliant. 1. Select Production Host Profile. Page 76

77 Not Compliant Notice that 'esx-05a' is Not Compliant against the Production Host Profile. Module 4, Challenge 3 Now that you've started down the correct path, it's time for you to take control. Your challenge is to: Resolve the issue so that 'esx-05a' is Compliant against the Production Host Profile. Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you want to see the solution to this challenge. Module 4, Challenge 3 Solution Were you able to resolve the issue with the Non-Compliant host? The Solution to this Challenge is as follows: 'esx-05a' is Non-Compliant due to an invalid NTP Server list. There is an additional NTP Server address within the NTP Server configuration of 'esx-05a' that is not located within the Host Profile. The NTP Server address 'ntp.corp.local' needs to be removed from the NTP Server list. Page 77

78 Take a Closer Look 1. Select Monitor. 2. Select Compliance. 3. Highlight the esx-05a.corp.local row. Do not select the 'esx-05a' object as it will take you to the host you selected. If this happens, select the Back button in the top right-hand corner of the vsphere Web Client to return to this page. 4. Notice the Host Value and the Host Profile Value fields are different. The Host Value has the additional NTP Server which is causing the host to be Not Compliant. Page 78

79 Modify the NTP Server Settings The NTP Server Settings can be modified within the vsphere Client (based on HTML5), therefore we will proceed with modifying the NTP Server Settings there. 1. Open a new tab within Google Chrome by selecting the reverse parallelogram. 2. Select Home. You will automatically be logged into vcenter since you are currently logged-in as CORP\ Administrator within the vsphere Flash Client. If you are not automatically logged-in, check the box to "Use windows session authentication" to login to vcenter. Hosts and Clusters 1. Select Menu. 2. Select Host and Clusters from the drop down list. Page 79

80 Time Configuration 1. Select esx-05a.corp.local. 2. Select Configure. 3. Select Time Configuration. 4. Select Edit. Edit Time Configuration Notice the additional NTP Server 'ntp.corp.local' within the NTP Servers text box. Delete 'ntp.corp.local' from within the text box. Check Host Compliance Page 80

81 Now that you have removed the invalid NTP Server entry, we need to re-check the Compliance of the host against the Host Profile. The vsphere Flash Client tab that you were working in within the first part of this lab should still be open. 1. Select the Check Host Profile Compliance icon to check the compliance of 'esx-05a' now that you have modified the NTP Server Settings. Compliant Great job. The host 'esx-05a' is now Compliant against the Production Host Profile. Your manager thanks you and your company can now move forward with standardizing their ESXi environment. If 'esx-05a' is still not compliant, there is no need to worry. The Hands-On Lab environment is delivered to you from many cloud environments. Due to this, sometimes the labs are created with slight differences from one cloud to another. Bonus Challenge: Examine the additional profile items that still make 'esx-05a' Not Compliant against the Production Host Profile. Edit the 'Production Host Profile' to make 'esx-05a' Compliant. If you're not interested in the Bonus Challenge, feel free to proceed to the next step. Stop Module 4 Before you celebrate with colleagues, open the Challenge Lab MS (Module Switcher) located on the desktop of the Main Console. Page 81

82 1. Double-click on Challenge Lab MS (Module Switcher) shortcut located on the desktop of the Main Console. 2. Select Stop under Module 4. Wait for the PowerCLI window to close before proceeding to the next step. You've Finished Module 4 Congatulations on finishing Module 4. You can now proceed in the manual to Module 5, or you can jump to a different module more suited for your skill level. Module 1 - Let's Start with the Basics (30 minutes) Module 2 - Basic Operations Management (30 minutes) Module 3 - Basic Resource Management (30 minutes) Module 4 - Advanced Operations Management (45 minutes) Module 5 - Advanced Resource Management (45 minutes) Page 82

83 Module 5 - Advanced Resource Management (45 minutes) Page 83

84 Introduction This module will challenge you on Advanced Resource Management tasks. My application is sensitive to latency, how do I ensure it gets the resources it needs to perform properly? I created a Resource Pool with the goal of improving performance, but performance is worse. What did I do wrong? I'm concerned my VM isn't configured correctly for vnuma, how can I check this? vsphere Client vs vsphere Flash Client In the vsphere Challenge Lab, you will use both the existing vsphere Web Client (based on Flash) and the new vsphere Client (based on HTML5). HTML5 is the future direction of the vsphere Client, so you will use it predominantly throughout this lab. If you feel more comfortable using the vsphere Web Client, a shortcut to it (labeled vsphere Flash Client) is bookmarked within Google Chrome. 1. The default client for the lab is the vsphere Client (based on HTML5). Use the Home button within Google Chrome to access the vsphere Client. 2. A bookmark to the vsphere Flash Client is available within Google Chrome. Note: As mentioned previously, the default client for this lab will be the vsphere Client based on HTML5. Due to the fact that the vsphere Client does not have full feature parity with the vsphere Web Client (based on Flash), there will be some sections where you will be directed to use the vsphere Web Client. Page 84

85 Module Switcher To prepare your lab for the Challenges, you will need to run the Challenge Lab MS (Module Switcher). 1. Double-click on the Challenge Lab MS shortcut located on the Main Console desktop. 2. Select Start under Module 5. Wait for the PowerCLI script to finish before proceeding. Once the script has completed, press Enter to close the PowerCLI window. Page 85

86 My Application is Sensitive to Latency ESXi is optimized for high throughput to minimize virtualization overhead for a widerange of applications, including business critical applications such as databases. Latency, measures the time it takes to transfer data from one endpoint to another and is typically measured in milliseconds. Latency-sensitive applications typically include VoIP services or high-frequency trading, where low response times are critical. More Information: Best Practices for Performance Tuning of Latency-Sensitive Workloads in VMware vsphere Open Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. If you did not Start Module 5 using the Challenge Lab Module Switcher outlined within the Introduction of this Module, please Stop and go back to the Introduction of Module 5 and follow the directions to Start the Module using the Challenge Lab Module Switcher. If you already completed this step, please proceed to the next page. Login to vsphere Flash Client The Latency Sensitivity feature is not available yet within the vsphere Client (based on HTML5), therefore you must use the vsphere Web Client to complete this challenge. 1. Select the vsphere Flash Client bookmark in Google Chrome. Page 86

87 Login to vsphere Flash Client 1. Check the box "Use Windows session authentication". 2. Select Login. Module 5, Challenge 1 Your Manager has informed you that the 'challenge-02a' virtual machine needs to have the Latency Sensitivity feature enabled on it. Your challenge is to: Enable Latency Sensitivity and ensure all Latency Sensitivity requirements are met for the virtual machine. Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you want to see The Solution to the Challenge. Module 5, Challenge 1 Solution To enable Latency Sensitivity on the 'challenge-02a' virtual machine, the following steps need to be completed. Right-click 'challenge-02a' > Edit Settings > VM Options > Advanced > Latency Sensitivity. Change the Latency Sensitivity setting from Normal to High. When changing the Latency Sensitivity setting from Normal to High, notice the warning you receive concerning CPU Reservation. Proceed to the next page to walk through this process step-by-step. Page 87

88 Edit challenge-02a 1. Right click on challenge-02a within the Inventory Tree. 2. Select Edit Settings. Page 88

89 Examine Virtual Hardware Examine the current virtual hardware settings for the 'challenge-02a' virtual machine, paying particular attention to the Reservation settings within CPU and Memory. 1. Select VM Options. Page 89

90 Latency Sensitivity Scroll down to Latency Sensitivity. 1. Use the dropdown to modify the Latency Sensitivity setting from Normal to High. Notice the Check CPU reservation warning. Select the information button to display a dialog box stating the following: For best performance, High Latency Sensitivity requires you to set 100% CPU reservation for this VM. 2. Select OK. Review Virtual Hardware Review the virtual machine virtual hardware settings to ensure the proper Reservations have been set. 1. Right click on 'challenge-02a'. 2. Select Edit Settings. Page 90

91 Review Virtual Hardware 1. Expand CPU and Memory. 2. Notice that setting Latency Sensitivity to High does not automatically set Reservations for CPU and Memory, therefore you will need to complete the process of enabling Latency Sensitivity on the 'challenge-02a' virtual machine. Page 91

92 Set Reservations 1. Select the CPU Reservation drop-down and select Maximum to configure the Maximum CPU Reservation for the virtual machine. For 'challenge-02a', that is 2880 MHz of CPU. 2. Select the Memory Reservation drop-down and select Maximum to configure the Maximum Memory Reservation for the virtual machine. For 'challenge-02a', that is 512 MB of Memory. More Information While you only receive a warning for CPU Reservation when setting Latency Sensitivity to High, it is highly recommended to set memory reservation to 100% as well. Without the memory reservation, vmotion will fail. If the virtual machine is powered-off, it cannot be powered-on until the reservation is set. Page 92

93 If multiple virtual machines are latency sensitive, spread them across hosts and tag them as Latency Sensitive. This will avoid over-utilization of hosts and results in better resource distribution. If you want to learn more about running latency sensitive applications on vsphere, consult these white papers: techpaper/latency-sensitive-perf-vsphere55-white-paper.pdf techpaper/vmw-tuning-latency-sensitive-workloads-white-paper.pdf This completes this challenge. Proceed to the next page to continue with Module 5 or use the Table of Contents to select a different challenge or module. Page 93

94 I Created a Resource Pool and now Performance is Worse Resource pools allow you to delegate control over resources of a host (or a cluster), but the benefits are evident when you use resource pools to compartmentalize all resources in a cluster. Using resource pools can result in the following benefits: Flexible hierarchical organization: Add, remove, or reorganize resource pools or change resource allocations as needed. Isolation between pools and sharing within pools: Top-level administrators can make a pool of resources available to a department-level administrator and any allocation changes that are internal to one departmental resource pool do not unfairly affect other unrelated resource pools. Access control and delegation: When a top-level administrator makes a resource pool available to a department-level administrator, that administrator can then perform all virtual machine creation and management within the boundaries of the resources to which the resource pool is entitled by the current shares, reservation, and limit settings. Delegation is usually done in conjunction with permissions settings. Separation of resources from hardware: If you are using clusters enabled for DRS, the resources of all hosts are always assigned to the cluster. That means administrators can perform resource management independently of the actual hosts that contribute to the resources. If you replace three 2GB hosts with two 3GB hosts, you do not need to make changes to your resource allocations. This separation allows administrators to think more about aggregate computing capacity and less about individual hosts. Management of sets of virtual machines running a multitier service: Group virtual machines for a multitier service in a resource pool. You do not need to set resources on each virtual machine. Instead, you can control the aggregate allocation of resources to the set of virtual machines by changing settings on their enclosing resource pool. Open Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. Page 94

95 Login to vsphere Client We will use the vsphere Client in this challenge. 1. Select the check box "Use Windows session authentication". 2. Select Login. FavoriteVMs Resource Pool Browse the Inventory Tree. Notice the 'FavoriteVMs' Resource Pool has already been created. Additionally, the 'challenge-01a' virtual machine has already been moved into the 'FavoriteVMs' Resource Pool. Page 95

96 Remote Desktop Open a remote desktop to both 'challenge-01a' and 'challenge-04a' using the shortcuts located on the Main Console desktop. Start Benchmarking Tool 1. A shortcut to the benchmarking tool, SPECjbb, is located on the desktop of both 'challenge-01a' and 'challenge-04a'. 2. Double-click on the SPECjbb shortcut to open the benchmark tool. 3. Select Start Benchmark on both 'challenge-01a' and 'challenge-04a'. 4. A widget has been placed on the desktop of both virtual machines that will allow you to monitor both CPU and Memory during the benchmark. Page 96

97 Module 5, Challenge 2 Modify the window size of the remote desktop sessions show that you can compare the performance results. Notice that 'challenge-04a' is performing better than 'challenge-01a'. Your challenge is to Determine why 'challenge-04a' is performing better than 'challenge-01a' and resolve the issue so that both virtual machines are performing equally. Need a Hint? You can also view hints on the VMware Technology Network Communities here: Page 97

98 DO NOT proceed to the next page unless you want to see the solution to the challenge. Module 5, Challenge 2 Solution Both 'challenge-01a' and 'challenge-04a' have the same resource configuration, therefore the issue must be related to the 'FavoriteVMs' Resourcee Pool. 1. Right-click on the FavoriteVMs Resource Pool in the Inventory Tree and select Edit Resource Settings. If you look closer at the 'FavoriteVMs' Resource Pool, you will notice that Shares have been configured for the resource pool. Since the 'challenge-01a' virtual machine is located within the 'FavoriteVMs' Resource Pool, that virtual machine does not utilize resources available to it, which includes the Custom Shares setting. 2. Select OK to close the Edit Resource Settings window. Page 98

99 Shares 1. Right-click on 'challenge-04a' and select Edit Settings. The 'challenge-04a' virtual machine has a Shares setting of Normal. Look closely at the Shares setting for 'challenge-04a' and you will notice a greyed-out Shares value of Toggle the Shares setting on 'challenge-04a' to Low and High. You will see the values changing based on the selected setting. Notice a setting of Low changes the Shares value to 1000, and a setting of High changes the Shares value to Do not modify the Shares setting at this time. 2. Select OK to close the Edit Settings window. Impact of Resource Pool Because the 'challenge-01a' virtual machine is located within the FavoriteVMs Resource Pool, it inherits the CPU Shares value of The 'challenge-04a' virtual machine has a Shares setting of Normal and after investigating further, the Normal setting has a Shares value of This would explain why 'challenge-01a' is performing roughly half as well as 'challenge-04a'. To resolve the issue you could do one of the following: Modify the CPU Shares setting of the FavoriteVMs Resource Pool to match that of 'challenge-04a'. In this instance, you would either change the CPU Shares setting of Custom to Normal or change the CPU Shares value from 1000 to You could remove 'challenge-01a' from the 'FavoriteVMs' Resource Pool. Page 99

100 You could modify the CPU Shares setting of 'challenge-04a' to match that of the 'FavoriteVMs' Resource Pool. In this instance, you would change the CPU Shares setting from Normal to Low or set a Custom value of However, with this solution, the performance of 'challenge-04a' would be cut in half, therefore this is not the recommended solution as your goal should always be to improve the performance of your virtual workloads. This completes this challenge. Proceed to the next page to continue with Module 5 or use the Table of Contents to select a different challenge or module. Page 100

101 vnuma Troubleshooting NUMA is becoming increasingly more important to ensure workloads, such as databases, allocate and consume memory within the same physical NUMA node that the vcpus are scheduled. When a virtual machine is sized larger than a single physical NUMA node, a vnuma topology is created and presented to the guest operating system. This virtual construct allows a workload within the virtual machine to benefit from physical NUMA, while continuing to support functions like vmotion. More Information: Virtual Machine CPU and vnuma Rightsizing - Rules of Thumb How ESXi NUMA Scheduling Works Open challenge-04a Remote Desktop If the 'challenge-04a' Remote Desktop session is not already open, double-click on the 'chg-04a.drp' desktop shortcut to open a remote desktop session to the virtual machine 'challenge-04a'. Examine NUMA 1. Double-click on the NUMA coreinfo shortcut located on the desktop of the 'challenge-04a' virtual machine. Page 101

102 Examine vnuma Notice that currently 'challenge-04a' is mapped to a single NUMA Node. You will also see the NUMA configuration using the tools provided by Microsoft Windows. Open Server Manager Coreinfo is a third party tool that has been installed on 'challenge-04a' make it easy to view the NUMA architecture. You can also natively examine the NUMA configuration on Microsoft Windows Server by performing the following: 1. Open Server Manager using the shortcut on the taskbar. Page 102

103 Open Resource Monitor 1. Select Tools. 2. Select Resource Monitor. Resource Monitor 1. Select the CPU tab. Page 103

104 2. Select the Views drop-down menu. 3. Select Small. 4. You can see that Windows 2012 detects two CPUs. There is no mention of NUMA nodes. vnuma As of vsphere 6.5, changing the corespersocket value no longer influences vnuma or the configuration of the vnuma topology. The configuration of vsockets and corespersocket only affects the presentation of the virtual processors to the guest OS (generally required for licensing). vnuma will automatically determine the proper vnuma topology to present to the guest OS based on the underlying ESXi host. For example, if you create a 4-vSocket virtual machine with 4 corespersocket (total of 16 vcpu) on a dual-socket, 16-core physical ESXi host, prior to vsphere 6.5, vnuma would have created 4 vnuma nodes based on the corespersocket setting. As of vsphere 6.5, the guest OS will still see 4 sockets and 4 cores per socket, but vnuma will now only create 1 vnuma node for the entire virtual machine since it can be placed in a single physical NUMA node. This new disconnection of the corespersocket setting with vnuma allows vsphere to determine the best vnuma topology automatically in all circumstances. To revert to the earlier behavior in vsphere 6.0, use the advanced setting: numa.followcorespersocket = 1 More Information concerning vnuma can be found in the following articles: techpaper/performance/whats-new-vsphere65-perf.pdf Module 5, Challenge 3 Page 104

105 In this challenge, the Database Manager is not comfortable with the new NUMA setting as it relates to the Database servers. Therefore, the Database Manager would prefer the NUMA behavior mimic that of vsphere 6.0. Your challenge is to: Modify the settings needed to create two NUMA nodes on 'challenge-04a'. Need a Hint? You can also view hints on the VMware Technology Network Communities here: DO NOT proceed to the next page unless you are ready to view the solution to this challenge. Module 5, Challenge 3 Solution If the Hot Add vcpu virtual hardware setting is enabled, then the architecture presented to the virtual machine will be uniform memory access (UMA) only. To solve this issue: 1. Power off the 'challenge-04a' virtual machine. 2. Within the vsphere Web Client, right click on 'challenge-04a', select Edit Settings. Within CPU, deselect Enable CPU Hot Add. 3. Select OK. 4. Power on the challenge-04a. Did the NUMA Architecture change on 'challenge-04a'? Additional steps are needed to force 'challenge-04a' to use 2 NUMA Nodes. We will complete the configuration of 'challenge-04a' in the upcoming steps. You should keep the Enable CPU Hot Add setting disabled unless you know that you need it. Note that the VM is configured with 2 vcpus and 1 cores per socket. Typically, you should leave Cores per Socket = 1 which results in the best performance. When Cores per Socket is left at its default value of 1, this allows ESXi to automatically choose the optimal vnuma configuration for the VM based on the physical server, which generally results in the best performance. Leave Cores per Socket = 1 unless you have a very specific reason to do otherwise, such as for licensing purposes. If your guest OS does not see a vnuma architecture and you think it should, check the following: Page 105

106 By default, vnuma is enabled only for virtual machines with more than eight vcpus. You can enable vnuma for virtual machines with eight or fewer vcpus by changing the VM setting numa.vcpu.min from its default value of 9 to the number of vcpus in the virtual machine. This setting controls the minimum number of vcpus required to enable vnuma. This setting was modified on 'challenge-04a'. vnuma requires virtual hardware version 8 or later. Other factors that may completely disable NUMA (for the host and the VM) are BIOS settings such as enabling node interleaving. Node Interleaving is generally disabled by default, but if you enable it, it will present a UMA architecture to the ESXi host. Open Google Chrome 1. Open Google Chrome using the shortcut on the taskbar. Open vsphere Flash Client 1. Select the vsphere Flash Client bookmark within Google Chrome. Login to vsphere Flash Client 1. Select the check box "Use windows session authentication". 2. Select Login. Page 106

107 Power Off challenge-04a Before proceeding to the next step, Power Off the 'challenge-04a' virtual machine. If 'challenge-04a' is already Powered-Off, you can proceed to the next step. 1. Right-click on 'challenge-04a'. 2. Select Power to expand the Power context menu. 3. Select Power Off. Page 107

108 Edit Settings 1. Right-click on 'challenge-04a' within the Inventory Tree. 2. Select Edit Settings. Page 108

109 Advanced VM Options 1. Select VM Options. 2. Expand Advanced. 3. Select Edit Configuration. Advanced VM Options The numa.vcpu.maxpervirtualnode Advanced Settings needs to be modified in order to enable multiple NUMA nodes on 'challenge-04a'. Change the numa.autosize.vcpu.maxpervirtualnode value to 1. If you don't modify this setting for 'challenge-04a', the virtual machine will fit within a single NUMA node and will be Page 109

110 presented as such. With the maxpervirtualnode value set at 1, we are forcing NUMA to recognize NUMA in multiples of 1, as opposed to 2 or 4. 2 NUMA Nodes Check the NUMA Node configuration after modifying the "maxpervirtualnode" setting to determine if a 2nd NUMA Node was successfully created on the 'challenge-04a' virtual machine. Use the desktop 'NUMA coreinfo' desktop shortcut once again to view the NUMA Node configuration. Great job, the 'challenge-04a' virtual machine can now take advantage of multiple NUMA nodes. Go to Server Manager > Resource Monitor and select the CPU tab. Notice that NUMA Node 0 and NUMA Node 1 are now visible within Resource Monitor. If you're curious to see if both NUMA nodes will be utilized, use the SPECjbb benchmark application located on the desktop to Start a Benchmark. View NUMA Node usage within Resource Monitor to ensure both NUMA Nodes are being utilized. If you DO NOT see 2 NUMA Nodes, be sure that you deselected Enable CPU Hot Add within the virtual machine settings. Page 110