TECHNICAL GUIDE. DataStream. Benchmarking Guide

Transcription

1 TECHNICAL GUIDE DataStream Benchmarking Guide Version February 2017

2 Table of Contents Introduction 3 Why Benchmark? 3 DataStream Architecture 4 Global Namespace (GNS) 4 Built-in Load Balancing 4 Tiering 5 Working Set 6 Scale-out Architecture 7 Benchmarking the DataStream Architecture 8 Benchmarking topologies 8 NFS VMkernel 9 Virtual IPs (VIPs) 9 Introduction to Iometer 10 Topology Panel 10 Disk worker 10 Disk Targets 11 Sectors 11 Starting Sectors 11 # of Outstanding I/Os 12 Access Specifications 12 General Benchmarking Advice 15 Indirect Hosts 15 Resources 16 Coho Data 16 Iometer 16 VMware 16 Other NA DataStream Benchmarking Guide

3 Introduction When considering any storage solution, an evaluation of its performance is going to be a key component. There are a wide variety of benchmarking tools available specifically to stress Enterprise Storage solutions such as the Coho Data DataStream. These tests can be complex to run and interpreting the results isn t always straightforward. This guide aims to provide guidance on selecting the most appropriate tests to run, why you should run such tests, and a brief step-by-step explanation of how to run some fundamental benchmark tests against a DataStream. Why Benchmark? Benchmarking tests are fundamentally designed to put pressure on a system. Tests can be designed to closely resemble genuine workloads, but to provide something which offers a close-enough approximation of a real environment can be extremely challenging. Simple testing of just one aspect of a system s performance is much easier to conduct and interpret, but is likely not realistic of how it would react in its intended role. There are many reasons to run benchmarking tests, and understanding which aspect of the system you are interested in should help guide the type of tests to run and how to configure them. For example, here are just some of the many valid reasons why you might want to run benchmarking tests: Compare to other solutions (for example a Proof of Concept bake-off). Quantify product performance improvement after new firmware/software. Determine if there are improvements after replacing or augmenting hardware. Discover storage system limits. Determine suitability for the current workload demands. Ascertain future growth while maintaining acceptable performance. Test for performance changes during other non-routine operations: component failures, adding capacity, migrating workloads, etc. Observe performance while using a particular feature; for example: a storage feature, such as DataStream SiteProtect site-to-site asynchronous replication, or some other infrastructure change such as a network reconfiguration NA DataStream Benchmarking Guide

4 DataStream Architecture Enterprise storage is not merely a sum of its hardware parts. Like any storage solution, Coho Data s DataStream architecture is unique and therefore behaves differently from other products in any given role. Coho Data has carefully designed its storage to provide outstanding performance, particularly in real-world workload scenarios. For the purpose of this guide, it's important to understand four key parts of the DataStream architecture: 1. Global namespace 2. Built-in load-balancing 3. Tiering 4. Scale-out architecture Global Namespace (GNS) Every DataStream chassis contains two storage controllers. Each controller has an independent namespace (for example, \\microarray1\, \\microarray2\,...). The DataStream global namespace (GNS) merges all the chassis namespaces into a single entity and presents a single virtual IP and MAC address, as well as a root mount point for vsphere clients. As more chassis are added to the cluster, the new controllers automatically become members of the same global namespace. This makes adding performance and capacity seamless. Built-in Load Balancing A key component of the DataStream architecture is the DataStream switch. Each cluster is deployed with a minimum of one (two for redundancy) Arista switch. The switch has two primary functions: 1. It provides a backend fabric for the nodes in a DataStream cluster. NFSD and other file system components use interfaces connected to this fabric to communicate with their counterparts in the cluster. 2. The DataStream switch provides a client-side storage network with a built-in load balancer for incoming NFS client connections. The load balancing aspect is achieved by a Coho Data developed OpenFlow controller. Incoming client connections are mapped to a node in the cluster based on the load and availability of cluster resources. After the initial connection is established, the load balancer steps out of the way and only interjects to rebalance client connects to ensure an even distribution of client loads across the cluster NA DataStream Benchmarking Guide

5 Tiering A common but often overlooked characteristic across all real-world workload types when doing synthetic benchmarking is that, in aggregate, not all data is hot. DataStream storage is specifically designed with this in mind. The DataStream architecture consists of a high performance NVMe (PCIe) flash tier, and a capacity tier consisting of either SATA SSD or SATA HDD; the performance is predominantly derived from the NVMe flash tier. When a new virtual machine is created, the entire virtual machine disk (flat.vmdk) resides in the upper flash tier. As the VM ages, infrequently accessed or cold parts of the flat.vmdk are demoted to the capacity tier in 512K blocks. For example, the figure below shows how a VMDK file might be split across the two tiers in a chassis. A VMDK Being Aged Across Tiers. Tiering ensures the availability of upper flash capacity for new writes and the reads of frequently accessed data. Tiering is also an efficiency-saving feature by holding only the data that requires the fastest flash performance in the the upper tier. By doing so, data with higher $/TB results in a much lower $/IOPS value NA DataStream Benchmarking Guide

6 Working Set The working set of a VM is the part of the flat.vmdk that is frequently read. Coho Data s FlashFit technology shows how much of the working set should be in PCIe flash. The curve of the graph indicates the percentage of the flat.vmdk that is being read from flash. This is called the hit ratio curve (HRC). In the example the hit ratio curve reaches a maximum of 10% of the entire flat.vmdk which is approximately 55 GB. The remaining cold portion of the flat.vmdk will be demoted to the capacity tier. Y axis: Read from flash percentage; X axis: Total size of working set. Benchmarking Impact As performance is derived from the upper flash tier, to generate benchmark results that simulate how real-world workloads would perform on a DataStream cluster, it makes sense to create workloads with real-world working set sizes that test the performance of the PCIe flash tier. Sample Workload Production VM: Win 8 64 Bit 2x 400 GB thin provisioned disks 250 GB used space per disk Total working set of 140 GB Benchmarking VM: Win 8 64 Bit Three disks (one for the OS, two for benchmarking) 2x 70 GB disks (represents working set of production vm) NA DataStream Benchmarking Guide

7 Scale-out Architecture When designing a storage solution, besides capacity planning, architects must also concern themselves with potential controller and fabric bottlenecks. Coho Data architecture is designed to avoid controller and fabric bottlenecks as a the cluster grows. The DataStream architecture is designed to scale without introducing bottlenecks at any level of the storage stack. That means every time a chassis is added, capacity and performance are added in a very linear fashion, at each level: Network Fabric Controller (compute) Performance Tier Capacity Tier Capacity and Performance Scale Linearly. To learn more about the architecture and scale-out capabilities, see the DataStream Architecture White Paper NA DataStream Benchmarking Guide

8 Benchmarking the DataStream Architecture The DataStream architecture is well equipped to handle any workload. The challenge is designing benchmark tests that measure this architecture with workloads that simulate real-world usage as closely as possible. Benchmarking topologies Benchmarks are often run in lab environments that inadequately simulate a production environment. For example: A. The Real-world Workload diagram outlines a real-world topology where read and write requests are coming from multiple guest VMs that are hosted on an ESXi cluster, with mounted NFS DataStream storage. B. The Poorly Designed Benchmarking Workload diagram outlines a typical benchmarking lab. A single VM, usually with a very large VMDK, is attempting to generate a large amount of IOPS. This unrealistic approach is prone to bottlenecks. (A) Real-world Workloads (B) Poorly Designed Benchmarking Workload NA DataStream Benchmarking Guide

9 A more effective solution is to duplicate the production environment as closely as possible: Try to estimate the working set size (files that will be in flash). Use a multiple IOmeter VMs with a sufficient amount of workers and unique disk targets. Use multiple 10 GB uplinks to connect hosts to DataStream switches. If possible, use multiple ESXi hosts or at the very least multiple VMkernel ports. The last two points can be difficult to achieve because most lab environments lack sufficient hardware for testing. If possible, create multiple VMkernel uplinks and additional DataStream VIPs to simulate NFS connections from a cluster of hosts. NFS VMkernel The VMware NFS VMkernel will send and receive traffic on a single uplink at one time. This means that even with multiple uplinks included in the NFS VMkernel port-group, only one uplink will be actively sending and receiving traffic at a given time. To generate more throughput from a single ESXi host as well as emulate an environment with multiple ESXi NFS clients, create additional VMkernel ports on the ESXi host. The VMkernel ports should use the same VLAN ID but be on separate subnets , , Virtual IPs (VIPs) As outlined in this guide, DataStream storage presents a single global IP (VIP) and MAC address through which the entire cluster is mounted by NFS clients. To accept NFS client requests from clients on a different subnets, additional VIPs must be added to the DataStream cluster via the DataStream API. Use the following curl command to view the current VIP: curl -- insecure - X GET - H 'Content-Type: application/json' - u admin :< datastream mgmt password > https : //<datastream-mgmt-ip>/api/mgmt-data/config/network/ grep vips -- VIP: NA DataStream Benchmarking Guide

10 To add additional VIPs: :~ $curl -- insecure - X PUT - H 'Content-Type: application/json' -- data '{"force": true, "vips": [" ", " , ]}' - u admin : <datastream mgmt password> https : //<datastream-mgmt-ip>/api/mgmt-data/config/network/storage/vips/ -- VIPS: , , It s worth noting that although this is useful to create more traffic in a lab, with a low number of hosts running a synthetic test, it s not required in normal usage. When using Coho Data storage, generally the number of ESXi hosts outnumber the number of DataStream nodes. Also, the VMs on each ESXi host are unlikely to generate enough IO to approach saturation on multiple 10 GB cables. Introduction to Iometer Iometer is an open source tool used measure performance under a controlled load. It is treated as an industry standard tool to test storage. For information on how to install and use Iometer see: After installing Iometer there are a few things to note: By default, the Iometer config file is found in C:\iomter\Iomter.icf. Do not run more than one instance of Iometer on a VM at a time. Topology Panel The first screen in Iometer is the topology panel. When Iometer is launched, a manager and a set of workers (threads) are automatically created. By default, Iometer will create one worker for every CPU core available to the virtual machine. All VMDKs added to the virtual machine appear as disk targets. Disk worker This is the thread that runs the actual workload. The number of workers is equal to the number of threads actually creating I/O requests. Each worker should be configured to one unique target. To increase the number of threads generating IO, increase the number of disk workers. The amount of host resources available to the Iometer VM dictates how many disk workers can be run on a VM. Adding too many workers per VM could result in CPU ready times on the guest VM and decreased IO NA DataStream Benchmarking Guide

11 Disk Targets Specifies the disks used by each worker (the actual VMDK). Alternatively, when a logical drive is created, a.tst file is written to the target. It's recommended that use VMDKs as disk targets instead of soft targets (.tst ). For physical disk targets (flat.vmdk), Iometer recognizes 2 volume types ( shown on next page ): Formatted disks (yellow icon with red line through it). Unpartitioned disks (blue icon). Iometer Physical Disk Targets. Use unpartitioned disks, as they can be used to test immediately and do not require a.tst file to be placed on the root directory of the disk. Use a unique disk target for every Worker. Sectors This represents the total target size, in sectors. The default 0 indicates the the entire flat.vmdk will be used. Alternatively you could create a large file (i.e. 100 GB), and calculate how many sectors would equate to a 35 GB working set. Calculation can be done using a unit storage conversion calculator. Starting Sectors This is the offset of the first sector, again this can be left at the default value NA DataStream Benchmarking Guide

12 # of Outstanding I/Os By increasing this number, the guest VM s OS will queue multiple IO requests. This helps saturate the storage and drive more IO. The outstanding number of IOs is typically set between 16 IOs/Target and 32 IOs/Target. Finding the perfect value requires some experimentation. As a general rule, start at 16 and increment by 4 until the total I/O number stops increasing. If the number of outstanding IO is too large, the Windows guest VM may crash. Access Specifications An interesting aspect of benchmarking is the obsession with IOPs. But high IOPS numbers does not necessarily mean high throughput or good performance. IOPS: input/output operations per second. Throughput: The amount of data that can be processed, expressed in MB/s. Latency: The amount of time it takes to complete an operation. Typically high IOPS values are produced by having a large number of clients making very frequent and very small requests to a small file. This is effective in producing hero benchmarking IOPS values but it is not an effective way of simulating a real-world workload. This example workload will produce high IOPS especially when run against PCIe flash. The unfortunate aspect about this workload is that it requires next to no throughput to produce. Kind of an empty win. File size Working Set Transfer Request Size Sequential Random Read Write 5 GB 100% 4 Kbytes Tests can be unintentionally constructed to produce meaningless results. The test below will produce immaterial lops values due to high latency induced by very large unrealistic requests: File size Working Set Transfer Request Size Sequential Random Read Write 100 GB 100% 1024 Kbytes A more real-world workload with a good balance between latency, iops, and throughput will look like the example below: File size Working Set Transfer Request Size Sequential Random Read Write 250 GB 25% 4 Kbytes Besides the preconfigured workloads, IOmeter provides a mechanism to create custom workloads with customizable access specifications NA DataStream Benchmarking Guide

13 Configurable parameters: Transfer request size: The number of bytes in each I/O request. Read/Write pattern: The percentage read vs the pertanage write. Access patterns: sequential vs percentage random. Iometer Access Specifications. Remember, as the transfer request size increases, the amount of time to complete an operation also increases, as a result of increased latency and a reduction in the number of operations performed in one second. It's important to understand this relationship. The tables below outline workload parameters to produce the maximum throughput numbers, maximum IOPs numbers and the lowest latency numbers. When applying these benchmarking parameters remember to size the workload as outlined in the Working Set section of this document. Workloads for maximum throughput Workload Rank (Best to Worst) Transfer Request Size Sequential Random Read Write 1st 64 Kbytes nd 128 Kbytes rd 128 Kbytes Table Outlines Workloads to Produce Highest Possible Throughput Numbers in Order of Effectiveness NA DataStream Benchmarking Guide

14 Workload for maximum IOPS Workload Rank (Best to Worst) Transfer Request Size Sequential Random Read Write 1st 4 Kbytes nd 8 Kbytes rd 16 Kbytes th 4 Kbytes th 8 Kbytes Table Outlines Workloads to Produce Highest Possible IOPS Numbers in Order of Effectiveness. Low Latency workloads Workload Rank (Best to Worst) Transfer Request Size Sequential Random Read Write 1st 4 Kbytes nd 4 Kbytes rd 8 Kbytes th 8 Kbytes Table Outlines Workloads to Produce Low Latency Operations NA DataStream Benchmarking Guide

15 General Benchmarking Advice Use the latest Coho Data DataStream version. Install the Coho Data VAAI plugin on all ESXi hosts that connect to the DataStream storage. Ensure that all ESXi hosts in a vcenter cluster have access to and mount the same storage. Update firmware on all ESXi servers. Use latest version of ESXi and vcenter Follow the advice in vsphere performance guide. Use Cat 6 cables with minimum length of at least 1 meter (4 feet). Indirect Hosts Coho Data strongly advises using the direct connect design for the absolute best performance results. However, this isn t appropriate in certain use-cases; e.g. blade servers. In such cases: Update firmware on any intermediary switch hardware. Follow intermediary switches vendor s best practises. Carefully consider the consolidation ratio. Monitor the port stats during benchmark test to watch for saturated ports. For guidelines on how to deploy intermediary switches see the following deployment guides, see the Resources section. Conducting benchmark tests on working storage will stress the infrastructure and may impact performance and availability of other workloads in the environment; this may include server and networking resources. Coho Data strongly recommends that all benchmark testing is conducted in an isolated lab environment to avoid affecting your working infrastructure NA DataStream Benchmarking Guide

16 Resources Coho Data Log into the Coho Data Customer Portal for the following recommended Coho Data resources: NA - Coho Data Quick Setup Guide NA - Coho Data Install Guide NA - Coho Data User Guide NA - Arista MLAG Reference Architecture NA - Cisco Nexus vpc Reference Architecture NA - Architecting DataStream Network Iometer Iometer documentation VMware VMware Best Practices for vsphere v6.x Other Unit storage conversion calculator. edited by SDC.. B<3 B Coho Data. All rights reserved. The Coho Data logo, DataStream, and MicroArray are trademarks of Coho Data in Canada, United States and other jurisdictions. All other trademarks, service marks, and trade names referenced in this document are those of their respective owners. No part of this document or other Coho Data document assets may be reproduced without the express written consent of Coho Data. Every effort has been made to ensure that the information in this document is accurate; however, errors and/or omissions in content are possible. In no event shall Coho Data be liable for incidental or consequential damages arising from use of this document or the software and hardware described in this document. Content is subject to change without prior notice. edited by SDC edited Shawn Connelly NA DataStream Benchmarking Guide