An Automation Framework for Benchmarking and Optimizing Performance of Remote Desktops in the Cloud

Transcription

1 217 International Conference on High Performance Computing & Simulation An Automation Framework for Benchmarking and Optimizing Performance of Remote Desktops in the Cloud Atul Pandey, Lan Vu, Vivek Puthiyaveettil, Hari Sivaraman, Uday Kurkure, Aravind Bappanadu VMware Palo Alto, USA {patul, lanv, vedatanputhiy, hsivaraman, ukurkure, Abstract In the current trend of moving everything into the cloud, cloud-based remote desktops are not an exception. Benchmarking virtual remote desktops for performance optimization is an important task for the successful development and deployment planning of virtual desktop infrastructure (VDI) used to deliver remote desktops. This task is very challenging at cloud scale because of rapid evolution of VDI software architectures with a very large number of remote desktops to be managed. In this paper, we present a new framework for evaluating VDI performance that has the capabilities of simulating real world VDI workloads and measuring important performance metrics at scale. Its design aims to provide facilities to easily automate the performance benchmarking tasks and the flexibility of adapting to changes in VDI software architecture, which are two major limitations of the existing solution. For evaluation, we present performance results of this framework. Keywords benchmarking, remote desktop, VDI, cloud computing, performance optimization I. INTRODUCTION Delivering remote desktops on the cloud to end-users using Virtual Desktop Infrastructure (VDI) technology [1], [2], [3], [4] has become more and more popular. With VDI, users can work remotely from anywhere and on any device like thin-clients, tablets, laptops, or even smartphones, and are able to access virtualized remote desktops running on datacenters of private or public clouds. Compared to the traditional physical desktop model, VDI provides centralized and simplified desktop management with multiple benefits like reducing the cost of software and hardware, enhancing mobility and security, and enabling fast adaptation to changes in technology. For example, doctors or health science researchers can work from home to access high resolution medical image data without worrying about security because the data is stored in datacenters [5, 6]. Architects/construction crews can work at construction sites and access the designs using remote desktops with high-end GPU support for 3D CAD applications instead of bringing their workstations to the site. Several popular VDI solutions include VMware Horizon, Citrix XenDesktop, and Microsoft VDI. Among them, VMware has been named the leader in recent years in this solution category [7, 8]. A key requirement for successful VDI adoption is ensuring the best end-user experience, while deploying it at scale. With the typical deployment scale ranging from hundreds to hundreds of thousands of remote desktops, optimizing and measuring VDI performance is very challenging because of (1) the complex architecture of a VDI system with multiple hardware and software components at both front end and back end, (2) the rapid changes in VDI technologies with continual increase in supported features and platforms, (3) performance impact of remote display protocols, and (4) the diversity of operating systems (e.g. Windows, Linux) and applications running in the remote desktops which can have different performance requirements (i.e. latency, memory, storage I/O, network, etc.) for different real use cases. At the same time, the existing performance benchmarking solutions [1, 2, 3] do not keep up with or fully support all the requirements of VDI evolution because of their limited functionality or incombability with newly release products. For example, [1] does not support pluggable module and multiple users; [1, 3, 4] do not support watermark measurement; [3] uses only Xen desktop and [1, 3, 4] do not support client driven workloads. In this paper, we present a new framework for benchmarking and optimizing the performance of remote desktops. This framework is an essential supplement to new, real requirements for optimizing VDI development and deployment at cloud scale. It also helps to evaluate VDI capacity planning and validate the scalability and user experience of new capabilities in VDI software development. It is designed to solve the challenges discussed above, and it allows quick adoption of evolving VDI technologies and user requirements. The framework provides a variety of API interfaces for performance benchmarking automation that existing solutions like LoginVSI do not possess. These API interfaces provide a powerful mechanism to rapidly build robust end-to-end automation that allows users to automatically deploy virtual desktops, install applications, run benchmarks/tests at scale, and gather the performance results. II. BACHGROUND A. Virtual Desktop Infrastructure VDI is virtualization technology that hosts a desktop operating system in a virtual machine (VM) on a centralized server in a datacenter. These virtualized desktops can have /17 $ IEEE DOI 1.119/HPCS

2 moderate computing power for regular users or high performance computing power with GPUs for special types of users like designers, scientists, etc. VDI employs a clientserver computing model where users use a client/web browser to connect to remote desktops running inside the datacenter to load its display content using remote display protocol and to interact with the desktop. Fig. 1 depicts an overview of VDI. Its software infrastructure can include multiple components like a VDI connection broker, VDI composer, load balancer, VDI agents, hypervisor, hypervisor manager, etc. Figure 1. A simplified architecture of VDI To deploy remote desktops at scale, VDI requires infrastructure with high network bandwidth, fast storage systems, servers with large CPU, memory resources, etc. To support 3D graphics applications, the servers can be equipped with multiple high-end GPUs. Each server can host hundreds of Windows/Linux desktop virtual machines and the total number of servers scale based on the number of real users. For large organizations like banks, hospitals, and higher education institutes, the scale of VDI deployment can go up to hundreds of thousands of remote desktops [6]. Therefore, optimizing the resource usage and increasing the consolidation of remote desktops per server is very important in saving costs. B. Remote Display Protocols Remote display protocol is a critical component of VDI. It connects end-user devices to a remote desktop or an application running remotely in a datacenter. The choice of remote display protocol has a huge impact on end-user experience and resource utilization in the datacenter. High quality user experience implies low latency, lossless images, high frame rates for videos, and requires higher network bandwidth and higher CPU utilization per user. VDI needs to strike a balance between resource consumption and end-user experience. VMware recently offers a new remote display protocol called Blast Extreme [9]. This protocol is based on the H.264 standard and can take advantage of the video encode and decode capability of GPUs offered by NVIDIA, AMD, and Intel to reduce the CPU consumption for longer battery life on mobile devices. Blast Extreme compensates for an increase in latency or a reduction in bandwidth. It leverages both TCP and UDP. When the desktop in the datacenter has access to a GPU, it flexibly offloads video encode/decode to the GPU, and this helps to improve the user experience. Otherwise, it uses the desktop s CPU for this purpose. Blast Extreme can be configured to deliver the desktop to the user without loss using PNG lossless format. Other remote display protocols include Teradici s PCoIP and Microsoft s RemoteFX which are supported by VMware VDI and other VDI products. C. Performance Benchmarking Requirements User experience is important in VDI. A realistic performance benchmarking tool has to accurately capture user experience metrics to evaluate if the VDI solution brings an end-user experience as seamless as on a physical desktop. In some cases, we expect VDI to deliver better user experience than the physical desktops (e.g. 3D graphics, machine learning, etc.) because the remote desktop has access to powerful computing resources in the datacenter. Depending on the type of applications running inside the desktop, the user experience metrics may include application response time, login time, frames per second, or total execution time, and measuring them accurately in a large scale distributed computing model of VDI is nontrivial. Scalability is another capability that a VDI performance benchmarking tool has to support. An efficient VDI solution needs to optimize the resource usage to achieve a high consolidation ratio (i.e. number of desktops per physical server) and scale well with the number of users. Frequently, performance issues are found only when the VDI deployment is tested at scale. In the cloud environment, this scale can be enormous; therefore, any useful benchmarking tool needs to have the ability to auto-scale the benchmark. This means that the benchmarking tool should start by deploying the benchmark at low scale and add virtual machines in steps until it detects a rapid increase in response time or until the response time crosses a pre-defined threshold value. The step size and the auto-scaling could be guided by simple queuing models or simple binary searches. Since VDI solutions, which typically require a large software stack, are still evolving rapidly, a benchmarking tool must be flexible enough to adapt or add components to handle changes in the VDI software stack. In summary, a useful benchmarking tool needs to be flexible with support for plugin modules, capable of auto-scaling, and support measurement of user experience in a VDI environment. III. A NEW FRAMEWORK FOR BENCHMARKING REMOTE DESKTOPS A. Benchmarking and Optimizing Remote Desktops VMware builds software to set up and manage datacenters. A key requirement to deliver quality software products with well-defined performance characteristics and behaviors is the ability to reliably run benchmarks and applications of interest on extremely large distributed systems composed of many thousands (or tens of thousands) of VMs running on different kinds of hardware. The software on which these benchmarks 746

3 need to run and the VDI itself are continually evolving due to better display protocols, just-in-time desktop delivery, and onpremise and cloud hosting platforms. Because of this, we need tools that can automatically and quickly provision or deploy the software infrastructure (such as VMware s VDI, AirWatch, or similar products), create as many VMs as needed, and then run applications or benchmarks at scale. Addressing this need, we build a general framework with a well-defined API to run applications/benchmarks at scale that has following core features: Figure 2. Remote desktop benchmarking model An object-oriented design that exploits the hierarchy of software components that define a datacenter to deliver services to the virtual machines in the benchmark or to the virtual machine that functions as the benchmark controller, called a harness in our framework. Ability to remotely run any command / executable on any virtual machine at any time. Capability of synchronizing among VMs running the benchmarks that requires two or more VMs working together as a team. The distributed benchmark itself would be run simultaneously on multiple such teams to generate the final benchmark score. The framework provides an out-of-band handshaking mechanism for the team of VMs to stay synchronized. Ability to disperse and collect data / files from all the components of the distributed system that participated in the framework. Our framework is currently being targeted for VMware s VDI solution. However, its design can be extended for benchmarking other distributed systems as well. The framework is not strongly tied with any external VDI component including VMWare products, each external component like VDI server, infrastructure manager and identity manager etc. are wrapped and abstracted with a generic API. Non VMware technologies can be wrapped and plugged in to framework, providing same capability as VMware. The working mechanism of this framework is demonstrated in Fig. 2 in which the green blocks are the components in our framework and the white blocks are VDI software components. A benchmarking job starts with the harness controller booting VMs at desktop side and client side. We use VMs to simulate end-user devices. Then, the client VMs automatically log in to the virtual desktops using VDI client or web browser. After login, the benchmarking workloads automatically start. At the same time, the performance tools implemented inside most components of the framework collect the performance data. When the workloads in all desktop VMs complete, the performance data is collected and stored in a database, and a performance report is generated. Depending on the real requirements, this process can be repeated multiple times with different test configurations specified by users or automatically created by other automation systems in the cloud for continuous performance monitoring. While many automation frameworks such as STAF can be used to run benchmarks at scale, none of them, to the best of our knowledge, possesses the following essential attributes that our framework has: Ability to deploy any applications or benchmarks without extensive scripting, i.e. a short turn-around time to obtain performance data. Our framework allows for arbitrary applications/benchmarks to be plugged in easily and run either in isolation or in tandem with other applications. This is extremely important for benchmarking in a cloud environment. Built in mechanisms for handshaking between VMs or containers that are co-operating closely to run the application. This is critical to be able to make sure that closely co-operating VMs in the benchmark stay synchronized. Without this ability, closely co-operating VMs would need to take recourse to time-outs to try to resynchronize in the event that they fall out of step with each other. Our experience shows that relying on timeouts is inefficient and error-prone. A built-in mechanism for the benchmark harness or controller to detect which VMs stop working. Ability to collect and consolidate essential performance metrics so that the benchmark controller can automatically determine to continue scaling the number of VMs in the benchmark runs. B. Architecture of the Benchmarking Framework The high-level architecture of the framework is described in Fig. 3. It is designed to meet following goals: 1) Multi-platform Support In order for the developed framework to support a diverse set of VDI products and platforms (i.e. Horizon View, Horizon Air, etc.), we build components in the framework as pluggable modules. 747

4 Figure 3. The architecture of the framework Infrastructure Manager: This pluggable module defines the API to interface with various infrastructure managers like VMware vcenter and cloud providers. This module provides infrastructure management services like VM power status, power cycle etc and infrastructure performance monitoring data like CPU, memory, and network use. This data is included in a test report and can be used for performance debugging and capacity planning. VDI Manager: This pluggable module defines the API for searching available desktop VMs for test, provides the required permissions to test domain users, and creates new desktop VMs based on test requirements. This module API is designed to interface and plug in to the management components of different VDI solutions. Identity Manager: In a real-world scenario, a user logs in to a remote desktop using domain credentials. This behavior is simulated using the Identity module. This pluggable module defines the API for creating and deleting new test users in a domain. This module can be used to plug in OpenLDAP or Microsoft Active Directory. 2) Scalable and Robust Infrastructure In a large VDI environment, thousands of users will be using various applications at a time. These users might belong to various categories like engineers, graphics designers, managers, etc. The usage pattern and application usage will be different for each kind of user group. Simulating these complex environments is key for realistic benchmarking and capacity planning. The following components of the framework have been designed to target these use cases: Perf Desktop Agents and Perf Client Agents: An agent is installed on each VM, which can work as either a desktop or client agent based on the role of the VM under test. The agent can synchronize with its paired VM to execute workloads. Its plug and play capability allows any application to be added as a workload. This agent provides timer and watermark services for latency measurement. An agent working as a client can simulate end users to log in to a remote desktop to trigger workload benchmarking. An agent working as the desktop agent can schedule and execute workloads assigned from the harness in random or round robin order. It also monitors and uploads performance and log data to the harness controller. Harness Controller: The Harness Controller provides the control logic for all other components. It orchestrates workload execution in the VMs by interacting with many external components like Hypervisor manager, VDI controller, Identity manager, VM agents, etc. Harness controller uses a well-defined API to abstract each external component, which makes it possible to support cloud and on-premises vendors by plugging in vendor-specific components. Harness control logic is also designed to be stateless; it uses a remote database for storing generated data. This capability allows us to wrap the harness logic in a container or VM image and create multiple harness instances at run time based on user load. o Message Router: This is a WAMP protocol-based router responsible for RPC communication between different modules. Each autonomous module registers with the router on initialization and exports its API. The router can register an API exported by connected modules and make them available for other connected modules. If a module goes offline during operation, the API supported by the component becomes unavailable for the moment. The API becomes available again when the module becomes accessible again. This feature is helpful when VMs need to restart during a test. o Harness Manager: This module takes care of authenticating user login and can also act as a load balancer and redirect the user login to an available harness instance. One harness can manage thousands of VMs with Perf Agents installed. For larger scale, multiple harness instances are deployed and managed by Harness Manager. 3) Diverse Workloads Workloads are the most important part of any VDI benchmarking tool. VDI use cases are diverse that makes the performance measuring requirements complex. Our framework addresses this challenge with the following features that allow any application to be automated and plugged in as a workload: Any application can be plugged in as a workload with a few API calls from a shared library that allows the workload to be implemented by any programing language. This is an advanced feature because a workload developer can choose any language in which to write the custom workload for a real use case. A plugged-in workload can register to a VM agent scheduler, execute various operations, and measure operation latency using timer and watermark services. 748

5 Custom application workloads generated by users are first class citizens and have the same capabilities as the predefined workloads that come with the framework. 4) End-to-End Automation A VDI benchmarking tool is not only used for benchmarking performance of various VDI products but also used to validate the performance of other datacenter infrastructure management software on which VDI runs. In a dynamic development environment, all of this software changes daily with new features continuously introduced. Hence, continuous performance testing with automation is a requirement. Our framework has following components to support this demand. Harness Interaction Module: This module manages harness administrative tasks via user interaction. Harness Interaction Manager communicates with the harness over WAMP router using a management API exported by the harness. This module has two variants: a GUI implementation for ease of use and a commandline module with scripting support. A script is just a collection of commands supported by the command-line management module and can be used to run completely autonomous tests on predefined time intervals. 5) Performance Measurement Methodology Accurately measuring performance metrics is an important and challenging role in benchmarking [1]. In a VDI environment, the latency of application operations is a good metric for which to evaluate the quality-of-service (QoS). Our framework allow measuring latency or response time of application operations at both server and client side. Both are important and complementary to each other. For measuring latency at the server side, the perf. agents component provides a pair of APIs, with which the workloads can mark the beginning and end of application operations. Based on these calls, the agent will log the operation name and the corresponding latency both of which will be processed by the harness at a later stage. For client-side latency estimation, an indirect method based on image watermarking is used. 6) Watermarking and Client-side Latency Measurement Digital watermarking is a well-known technique used for augmenting multimedia contents with additional information without perceivable distortion to the original content [11]. Our framework uses a variant of image watermarking for passing information about the application operations from server to client, which in turn will be used at the client-side to capture and log the operation latencies. A predefined rectangular region on the server s display is used as watermark window where the data to be communicated to the client will be displayed as a pixel pattern, which can be read back and interpreted from the client. The method used for encrypting bits into pixel patterns tries to be resilient against pixel-level distortions, which are caused by the compression algorithms, by providing redundant data. In the implementation, a 2x4 grid of RGB pixels is used for encrypting each bit of data, with a white grid representing the value 1 and a black grid representing the value. To add another level of redundancy, a pixel pattern corresponding to the inverse of the original data is also placed inside the watermark window, which is used for data validation at the client side. Fig. 4 illustrates the encryption at the server side with a synthesized image generated for a binary sequence "111". At the client side, a thresholding and majority votingbased algorithm is used to decrypt pixel grids to corresponding bit values, which is depicted in Fig. 5 for the binary sequence "111". If the data decrypted from the watermark region is found to be inconsistent, it will be discarded. Figure 4: Synthesized watermark image for sequence "111" Figure 5. (a) Watermark region on client display for sequence "111". (b) After thresholding and voting-based transformation To make the watermarking framework flexible to be used by any generic workload, the VM Agent component provides a pair of APIs with which a workload can mark the beginning and end of any temporal event (application operation) and assign a name to it. The agent, in turn, will assign a unique numerical ID (16-bit) to any such event at its beginning, and this ID with a corresponding begin/end marker (1-bit) will be communicated to the client through the watermark. At the server-side, the mapping between ID and operation name provided by the workload will be maintained; whereas, on the client side the ID and corresponding latency will be logged. The new tool also supports workloads with built-in watermarks, mainly to support Video workloads targeting frame rate measurement. In such cases, the location of the built-in watermark is expected to be provided in the workload configuration. 7) Capacity Planning with Resource Utilization The ability to simulate multiple user groups with different application use patterns and flexibility to plug in new workloads makes the new tool a suitable platform for realistic capacity estimation and planning. Though the QoS assessed with application latency is quite useful in benchmarking a VDI deployment, it provides limited insight into the infrastructure capacity. To overcome this limitation, the tool collects and reports data about the utilization of host resources such as CPU, memory, and storage and network interfaces during the benchmark cycles. This is done with the help of an Infra-management component of the tool. In addition to being useful for capacity estimation, the patterns in the resource utilization data can be quite effectively used in root cause analysis of badly performing VDI deployments. 749

6 IV. PERFORMANCE RESULTS In this section, we illustrate some capabilities of the presented framework via performance results of different application sets. A. Benchmarking Regular Applications on Remote Desktops Our benchmarking tool includes a typical office user workload, a robust technique to accurately measure end-to-end latency, and a virtual appliance to configure and manage the workload run at scale. Ideally, the workload needs to incorporate the many traditional applications that typical desktop users use in the office environment. These applications include Microsoft Office applications (Outlook, PowerPoint, Excel, Word), Adobe Reader, Windows Media player, Internet Explorer etc. Here, we present benchmarking results of these applications as a standard benchmark of our tool. The challenge lies in simulating the operations of these applications so they can run at scale in an automated and robust manner. Additionally, the workload should be easily extensible to allow for new applications, and be configurable to apply the representative load for a typical end user. We discuss these challenges and illustrate how we solved them to build a robust and automated workload, as described in Section III. 1) User Experience As mentioned earlier, user experience defines the success of a VDI solution. Fig. 6 shows the user experience metrics measured and reported by our framework for the standard benchmark. They include Quality of Service (QoS), login time, frame per second (FPS), and operation latencies. Fig. 6a, 6b, and 6d are the results of benchmarking standard workloads with 1 Windows1 desktop VMs using VMware's Horizon View 7.1 and App Volumes 2.12 on Dell R73 servers with dual 12-core Intel Haswell sockets, 26 GB RAM, and allflash SSD array storage. Fig. 6c presents the benchmarking performance of the video workload with Horizon View 7.1 on Dell R72 servers with dual 14-core Broadwell Intel sockets, 768 GB RAM, and SSD storage. This video benchmark used NVIDIA M1 GPUs (M1-1b profiles) for accelerating the Blast Extreme remote display protocol. Each workload of the standard benchmark is a set of operations performed on a certain application. For example, with MS Word, the workload opens the application, creates a Word file, edits it, saves it, and closes the application. When a benchmark runs, a series of operations are executed and their end-to-end latencies are measured using the watermarking technique. A typical standard benchmark run consists of over 5, operations of over 4 operation types. These types of operations are categorized into 3 groups: Group A includes interactive operations like web browsing and minimize/maximize applications. Group B includes I/O operations like open and save file. Group C includes background load operations. QoS, determined separately for Group A user operations and Group B user operations, is the 95th percentile latency of all the operations in a group. The default thresholds are 1. seconds for Group A and 6. seconds for Group B. We do not Figure 6. User Experience Metrics 75

7 recommend the configuration with a latency that is larger than these thresholds. Group C represents long running background operations so there is no time threshold defined for it. 2) Consolidation and Scalability Identifying consolidation ratio, scale, and resource bottlenecks is frequent requirement in VDI development and deployment. Our framework is a helpful tool for this purpose. Consolidation and scalability are two concepts usually used in VDI deployment. The first one is the maximum number of desktop VMs per server while QoS is maintained and identified while scaling the number of VMs per server. The other is scaling the number of servers on data center. In addition to user experience metrics, tens of different resource utilization metrics (i.e. CPU, GPU, memory, storage, network, etc.) are also measured and reported to provide comprehensive data for performance analysis and optimization. We present here only a few highlighted metrics. 95th percentile latency (sec.) (a) Scalability Group A Group B 1 5 # of VMs 95th percentile latency (sec.) (c) CPU utilization (%) (d) Memory utilization (%) # of VMs # of VMs Figure 7. Scalability and consolidation For example, Fig. 7b, 7c, and 7d help to specify the maximum consolidation of Windows 1 desktops (running on Dell R73 servers with dual 12-core Intel Haswell sockets and 26 GB RAM) for regular users, which is 12 VMs with Horizon View 7.3 and App Volumes For a larger number of VMs, QoS failed due to the group B latency being larger than 6 seconds a threshold that we use as an cut-off limit for group B and the CPUs being saturated. This ratio changes for different deployment configurations. We illustrate the scalability of our framework with a test of 5 Window 7 desktop VMs with Horizon View 7.3 running on 4 Supermicro servers with dual 8-core Intel sockets, 256 GB RAM, and SSD storage in Fig. 7a. The performance results in this test show both consolidation and scalability can be slightly increased because the QoS is still met. B. Benchmarking 3D Graphics Applications & Virtual GPUs VDI users who are CAD designers benefits from the mobility and cost efficiency of remote desktops that share (b) Consolidation Group A Group B # of VMs high-end GPUs using different virtual GPU solutions like vgpu. Benchmarking 3D graphics applications also requires monitoring QoS delivered by the virtual GPUs with different vgpu configurations, called vgpu profiles. It also requires additional metrics like encoded FPS, delivered FPS, and GPU utilization. Our framework has the capability of plug-and-play workloads, making benchmarking such applications easy. 3D graphics benchmarking can be used to evaluate different virtual GPU solutions. Avg Frame per Second (a) FPS per VM for different vgpu profiles M6-8Q (1VM) M6-4Q (1VM) M6-2Q (1VM) vgpu profiles M6-1Q (1VM) (c) CPU Utilization (%) M6-8Q (4VMs) M6-4Q (6VMs) M6-2Q (12VMs) vgpu profiles M6-1Q (24VMs) Avg Frame per Second Figure 8. 3D graphics benchmarking with SPECapc for 3DSMax 215 Fig. 8 illustrates the results of benchmarking with SPECapc for Autodesk 3DS Max 215 on Windows 7 desktop VMs running Dell R72 servers with dual 14-core Broadwell Intel sockets and 4 NVIDIA M6 GPUs, 768 GB RAM, and SSD storage. We present in Fig. 8a the average FPS measured for different vgpu profiles where each defines the amount of device memory a virtual GPU has (i.e. M6-1Q with 1 GB VRAM, M6-2Q with 2 GB VRAM, M6-4Q with 4 GB VRAM, M6-8Q with 8 GB VRAM). Fig. 8b, 8c, and 8d captures the performance metrics of the same benchmark on VMs with M6-1Q vgpu when the number of desktop VMs increases. Each vgpu profile shows a different density of 3D graphics desktops that a server supports because of the memory size of vgpu profile. C. Benchmarking High Performance Computing Applications Users with remote desktops with large CPU, memory resources, or even with GPUs can also run high performance computing applications like machine learning, Monte Carlo simulation, etc. In such cases, our framework design allows a mechanism to perform such customized workloads, as well as collect and aggregate the performance data. Here we discuss the performance of several machine learning (ML) workloads that can be run using our benchmarking framework. In our experiment, we used a set of three ML applications including (1) a language modeling using recurrent neural network (RNN) on the PTB dataset [13, 14], (2) handwriting (b) FPS vgpu M6-1Q Number of Virtual Machines (d) GPU Utilization (%) M6-8Q (4VMs) M6-4Q (6VMs) M6-2Q (12VMs) vgpu profiles M6-1Q (24VMs) 751

8 recognition using CNN with MNIST dataset [12, 15] and (3) object recognition using CNN with CIFAR-1 dataset [16]. These applications are implemented using Tensorflow [17]. We tested two cases: (1) comparing performance of GPU vs. without GPU with language modeling and handwritten recognition neural network, (2) scaling the number of GPUs with object recognition. The applications were run inside a Linux-based virtual machine with 12 virtual CPUs, 6GB RAM, and 96GB SSD storage on VMware vsphere 6 hypervisor. The test server was Dell PowerEdge R73 server with dual 12-core Intel Xeon CPU E5-268 v3, 2.5 GHz sockets (24 physical core, 48 logical with hyperthreading enabled), 768 GB memory, SSD (1.5TB). This server also has two Nvidia Tesla M6 cards (each has two GPUs) for a total of 4 GPUs where each has 248 CUDA cores, 8GB memory. Upon the results of first test case are shown Fig. 9a and 9b, we find a performance gain with GPU 7.9 times for language modeling and 1.1 times for handwritten recognition. Normalized Training Time (a) Language Modeling with RNN on PTB With GPU Without GPU Figure 9. Performance comparison with and without GPUs (a) Normalized Images / Sec # of GPUs Normalized Training Time (b) Handwritten Regconition with CNN on MNIST 15 Figure 1. Scaling with # of GPU on object recognition on CIFAR-1 The results of the second test case (Fig. 1) show the scalability of ML applications with GPU and the capability of our framework in benchmarking HPC workloads with different number of GPUs per VM. As the number of GPUs is increased, we see higher normalized images per second processed. The metric images processed per second has direct impact on training times. The metric images processed / second can be interpreted as training examples processed per second. CPU utilization is also tracked as the number of GPUs used is increased. This helps us size the VM to maximize processed images/sec for resources utilized as shown in Fig. 1d. V. CONCLUSION We present our framework for performance benchmarking remote desktops in the cloud that meets the real 1 8% 6% 4% 2% % 5 1. With GPU 1.1 Without GPU (b) CPU utilization # of GPUs requirements of VDI solutions. This framework has advanced characteristics: (1) support for multiple VDI solutions like Horizon View for on-premises or Horizon Air for public cloud environments; (2) design that easily adapts to the rapid evolution of VDI software by providing scalability and robustness; (3) support of diverse workloads with the plugand-play mechanism; (4) standardization of API support, which allows end-to-end automation. Finally, we demonstrate some capabilities of the framework with the performance benchmarking results for different types of applications, different hardware and software platforms, and with various scaling. In future, we plan to add the analytics component into our framework to automatically provide performance analysis and optimization recommendations. ACKNOWLEDGMENT The authors would like to thank Julie Brodeur for her help with our paper. REFERENCES [1] A. Berryman, P. Calyam, M. Honigford, A. Lai, "VDBench: A Benchmarking Toolkit for Thin-client based Virtual Desktop Environments," 2nd IEEE Int. Conf. on Cloud Computing Technology and Science, pp , 21. [2] B. Agrawal, L. Spracklen, R. Bidarkar, U. Kurkure, S. Satnur, V. Makhija, T. Magdon-Ismail, "VMware View Planner: Measuring True Virtual Desktop Experience at Scale," VMware Technical Journal, 29. [3] S.Y. Wang, W.J. Wang, "Benchmarking the Performance of XenDesktop Virtual DeskTop Infrastructure (VDI) Platform," in Proc. of the 14th Int. Conf. on Networks, pp.37-43, 215. [4] P. Casas, M. Seufert, S. Egger, R. Schatz, "Quality of Experience in Remote Virtual Desktop Services," in Proc. of 213 IFIP/IEEE Int. Symposium on Integrated Network Management, pp , 213. [5] Top 5 Use Cases for VDI, [6] VMware AlwaysOn Point of Care Solution Reference Implementation Case Study for European Healthcare Provider, Technical White Paper, 212 [7] R. Young, D. Laing, "IDC MarketScape: Worldwide Virtual Client Computing Software 215 Vendor Assessment," IDC MarketScape, 215. [8] R. Young, D. Laing, "IDC MarketScape: Worldwide Virtual Client Computing Software 216 Vendor Assessment," IDC MarketScape, 216. [9] C. Arakelian, C. Halstead, "Blast Extreme display protocol in Horizon 7," in VMware technical white paper, Oct [1] R. Jain, "The Art of Computer Systems Performance Analysis," John Wiley and Sons, New York, [11] C. I. Podilchuk, E. J. Delp, "Digital watermarking: algorithms and applications," in IEEE Signal Processing Magazine, vo. 18, pp , August 22. [12] U. Kurkure, L. Vu, H. Sivaraman, "Machine Learning on VMware vsphere 6 with NVIDIA GPUs," in VMware blog, Oct [13] W. Zaremba, I. Sutskever, O. Vinyals, "Recurrent Neural Network Regularization," In arxiv: , 214. [14] A. Taylor, M. Marcus, B. Santorini, "The Penn Treebank: An Overview," Anne Abeille, TreeBank, Kluwer, 23. [15] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, "Gradient-based learning applied to document recognition," in Proc.of the IEEE, 86(11): , November [16] A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images," [17] 752