Technical Note TN1070: Verint Master Server Under VMware 6.0 Publication Date: Monday, May 23, 2016 Products Exposure Level Technical Note Type Description Verint VMS 6.4 SP3 and higher VMware 6.0 Customer Facing Product Integration This technical note describes the recommendations for integrating VMware solutions with the Master Server in a High Availability deployment, with a reduced IT footprint. It also offers troubleshooting information for performance and ready time. Unauthorized use, duplication, or modification of this document in whole or in part without the written consent of Verint Systems Inc. is strictly prohibited. By providing this document, Verint Systems Inc. is not making any representations regarding the correctness or completeness of its contents and reserves the right to alter this document at any time without notice. Features listed in this document are subject to change. Contact your Verint representative for current product features and specifications. All marks referenced herein with the or TM symbol are registered trademarks or trademarks of Verint Systems Inc. or its subsidiaries. All rights reserved. All other marks are trademarks of their respective owners. 2016 Verint Systems Inc. All Rights Reserved Worldwide.
Introduction Introduction VMware High Availability (HA) provides easy-to-use, cost effective high availability for applications running in virtual machines. In the event of physical server failure, affected virtual machines are automatically restarted on other production servers with spare capacity. In the case of operating system failure, VMware HA restarts the affected virtual machine on the same physical server. VMware HA allows IT organizations to: Minimize unplanned downtime and IT service disruption, while eliminating the need for dedicated standby hardware and installation of additional software. Provide affordable, uniform high availability across the entire virtualized IT environment, without the cost and complexity of failover solutions tied to either operating systems or specific application. Prerequisites The environment must meet the following software and system prerequisites: VMware ESXi 6.0 U2 VMware vcenter Server 6.0 U2 Microsoft Windows Server 2016 R2 (guest operating system running under ESXi) Cluster configuration with a minimum of two ESX hosts running, enabling HA feature. Shared storage (iscsi or FC) configured and recognized for all ESX hosts within the cluster. Additional Information For more information about using VMware 6.0 in general, and specifically in a High Availability environment: Performance Best Practices for VMware vsphere 6.0 VMware High Availability 2 2016 Verint Systems Inc. All Rights Reserved Worldwide.
Hardware Requirements Hardware Requirements It is highly recommended that the servers, storage adaptors, and other hardware you are using in the integration are officially supported by VMware. For the list of supported hardware, see http://www.vmware.com/resources/compatibility/search.php. Note The recommendations for running Verint VMS under a VMware environment are different from those for running Verint VMS without VMware. The following table describes the hardware requirements of a system where VMs will be running in a VMware vsphere environment. Component Specification CPU Intel Haswell generation (such as Intel Xeon 2600-v3 Series) or later Memory Network Storage Recommendation 6% More than total running VMs VM Network (VMS LAN Network) VMKernel Network (vmotion) Virtual Datastore on iscsi, SAS or FC Supported by VMware for the VM, supporting 3000 IOPS Configure a dedicated vswitch for VMKernel (ensure that the vswitch is not shared between the VMKernel and Virtual Machine port groups). Virtual Machine Requirements The specifications for the virtual hardware are equivalent to the recommended specifications for the Master Server described in the Verint VMS Customer Furnished Equipment Guide. This section describes the requirements for memory, network and storage for Virtual Machines (VMs) in a High Availability environment. Component vcpu Memory Network Recommended Specifications 8 vcpu (HA) 8 GB of RAM VMXNET3 Storage 3 partitions (C, D and E) A minimum of 150 GB (3 x VMDK disks), where partition E (which hosts the SQL Server data files) is aligned to 64K 2016 Verint Systems Inc. All Rights Reserved Worldwide. 3
Storage Requirements Component Recommended Specifications Storage performance: 1500 IOPS (up to 1000 cameras) 3000 IOPS (up to 2000 cameras) Virtual Machine Configuration Observe the following guidelines when configuring the VM: Reserve 1200 MHz for each VM. Keep the average CPU utilization of the host machine below 75% of the total CPU usage, to ensure that the guest VMs run properly. VMware Tools automatically synchronizes the time of the guest operating system to match the host operating system, even if periodic clock synchronization is turned off. Therefore, the following components should sync with an NTP Server: All ESX hosts All Windows guest OSs (Master Server) Storage Requirements This section describes the available Storage Area Network (SAN) options that can be used with the Verint Recorder and ESM servers. Fibre Channel SAN: It is best to use a Fibre Channel SAN when ESXi is hosting any number of Verint Recorder or Verint ESM servers. The recommended minimum Fibre Channel speed is 8Gbps. iscsi SAN: A typical legacy iscsi SAN provides lower performance than a Fiber Channel SAN because it generally uses a 1Gbps Network Interface Card (NIC). When using an iscsi SAN in an HA deployment, it is recommended to use a 10Gbps NIC and enable jumbo frames. Recommendations Use a Fibre Channel SAN whenever possible. For the best storage performance, use VAAI-capable storage hardware. 4 2016 Verint Systems Inc. All Rights Reserved Worldwide.
Troubleshooting Troubleshooting This section describes how to identify and fix problems in a noticeably degraded Verint VMS system that uses VMware. It includes procedures for identifying performance issues due to overloading of the virtual machine s host. A basic understanding of VMware is required in order to perform the troubleshooting tasks in this section, such as how to use the vsphere Client and the vcenter. The following troubleshooting information is included: Monitoring performance Assessing CPU ready time Monitoring CPU Performance The vsphere Client can be used to identify problems on both the physical host machine and the VM. In the vsphere Client application, there are two important symbols to pay attention to, the Yellow Caution Alert and the Red Alert. These symbols indicate a problem that must be either resolved or monitored. They appear when a threshold that has been defined in a rule is surpassed, with or without a condition. Examples include sending a warning if the CPU exceeds 85% for five minutes, and sending an alert if the CPU reaches 100% for five minutes. To monitor CPU performance 1. Open the vsphere Client. 2. Do one of the following: Highlight the ESX host for general real-time data capture. Highlight any specific VM for more accurate data. 3. Click the Performance tab. The Real-Time Graph is displayed. 2016 Verint Systems Inc. All Rights Reserved Worldwide. 5
Monitoring CPU Performance For troubleshooting purposes, the Real-Time Graph can be exported as an image, or to a Microsoft Excel file with all the currently displayed data. This can help track multiple time slots in order to analyze if CPU or memory use are growing with time. 4. (Optional) Click Chart Options to specify a range of dates, objects and counters to be displayed on the chart. For example, choose the option Past Year with Object 1, and Object 2 for CPU Core 1 and 2, and the counter Usage as an average. Then, export the results for further analysis. 6 2016 Verint Systems Inc. All Rights Reserved Worldwide.
Assessing CPU Ready-Time Status Assessing CPU Ready-Time Status To ensure that the system is properly installed and configured, you need to confirm that the CPU ready-time status of the virtual machines is appropriate. CPU ready time is the time that a VM waits in a ready-to-run state before being scheduled on a CPU. A low ready time is acceptable and does not noticeably affect performance. The higher the ready time, the more it will affect VM performance. This section describes the following: Identifying a High CPU Ready Time Optimizing the CPU Ready Time Verifying Total CPU Use 2016 Verint Systems Inc. All Rights Reserved Worldwide. 7
Identifying a High CPU Ready Time Identifying a High CPU Ready Time The CPU ready time can be affected by any of the following: Busy ESX console Sharing resources (memory, CPU, disk, and so on) Large number of VMs compared to hardware availability for overall CPU utilization A high CPU ready time can cause the following problems: A slow VM Loss of communication with the VM, causing network failure or bad data transfer To assess the CPU ready time on the host machine, you need a direct console connection to the ESX host or a SSH connection, using a tool such as Putty. Perform the following procedure after you are connected to the console and authenticated. To identify a high CPU ready time 1. Log in to the ESX host that contains the VM using an SSH connection. Note Ensure that the ESX host is enabled for SSH remote connections. 2. In the console, type esxtop. 3. Press c on the keyboard. 4. Look at the %RDY column for the VMs and the console. 8 2016 Verint Systems Inc. All Rights Reserved Worldwide.
Optimizing the CPU Ready Time Ready Time Parameters Load Average PCPU CCPU Description This is the use for all physical CPUs. It displays for one-, five-, and fifteen-minute intervals Percentage of the individual physical CPU Percentage of total CPU time: us: % User Time sy: % System Time id: % Idle Time wa:% Wait Time cs-sec: Context switch per second The lower the percentage, the better the CPU ready time. A low percentage indicates that the virtual machine did not waste time waiting for the next CPU cycle. 0 to 5% is required 5 to 10% is acceptable but leaves room for improvement 10% + is not acceptable If the percentage is too high, it most probably means that the system is overloaded with virtual machines, and/or sharing too many resources. Methods for preventing a high CPU ready time include CPU reservation, and setting the affinity to protect the ESX console from overload. The total CPU use for the host should be below 75%. Optimizing the CPU Ready Time This procedure should be implemented to prevent VMs from using CPU processing from the ESX console. This ensures that the console is not busy scheduling CPU time for the VMs. The console always uses affinity 0 to operate. When performing this procedure, you need the number of CPUs on the physical host in order to know how many VMs to configure. For example, if the host machine has 8 cores, the maximum configuration is 4 VMs. Each VM requires 2 dedicated CPUs. To optimize the CPU ready time 1. Right-click a VM in the vsphere Client. 2. Select Edit Settings. 3. Select the Resources tab. 4. Highlight CPU and set the Reservation value to 1200 (MHz) or higher. 2016 Verint Systems Inc. All Rights Reserved Worldwide. 9
Verifying Total CPU Use Verifying Total CPU Use It is very important to ensure that the VMs are not overloaded with CPU tasks. To verify the total CPU use 1. Select the ESX host in the vsphere Client. 2. Select the Performance tab. A graph is displayed showing the real-time activity for CPU, memory, disk. 3. Verify that the Average Percentage on the host and all the cores does not exceed 75% of the total CPU. If the Average Percentage is above 75%, consider adding more ESX hosts to your data center to balance the load more effectively. 10 2016 Verint Systems Inc. All Rights Reserved Worldwide.