Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 VIRT1445BU Extreme Performance: Fast Virtualized Hadoop and Spark on All-Flash Disks VMworld 2017 Dave Jaffe, Performance Engineering, VMware Justin Murray, Technical Marketing, VMware Content: Not for publication #VMworld #VIRT1445BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #VIRT1445BU CONFIDENTIAL 2

3 Agenda 1 Speaker Introductions 2 Review of Big Data Architecture 3 Introduction to the Performance Area 4 Test Configurations 5 Workloads 6 Performance Results 7 Best Practices 8 Tuning 9 Overview of Machine Learning 10 Conclusions #VIRT1445BU CONFIDENTIAL 3

4 Our Roles Dave is an engineer on the performance team at VMware, focusing on Big Data. Justin is in the Technical Marketing area at VMware, where he provides technical information to partners and customers who are deploying big data systems on vsphere or distribution #VIRT1445BU CONFIDENTIAL 4

5 Why the Customer Interest in Big Data? Want to get off existing costly data platforms Older data warehouse technology is not serving our needs Want to do queries and analytics against many different forms of data (structured, unstructured, streaming) Provide data access to our customers Integrate systems that have been islands till now Single source of truth for the enterprise Exploit new application architectures for developer productivity Want to do data science, machine learning, deep learning #VIRT1445BU CONFIDENTIAL 5

6 The Existing Hadoop Architecture Client ResourceManager Master Scheduler NameNode submit job Master File System Index Worker Node 1 Worker Node 2 Worker Node 3 Nodemanager AppMaster - 1 Datanode Workers Nodemanager Datanode Nodemanager Container - 2 Container - 3 Datanode HDFS Block 1 HDFS Block 2 HDFS Block 3 #VIRT1971QU CONFIDENTIAL 6

7 High Level View of Apache Spark #VIRT1445BU CONFIDENTIAL 7

8 The Spark Architecture Standalone Driver Job Worker Node 1 Worker Node 2 Worker Node 3 Executor JVM Executor JVM Executor JVM Executor JVM Executor JVM Executor JVM #VIRT1445BU CONFIDENTIAL 8

9 Spark Implemented on YARN Job Worker Node 1 Worker Node 2 Worker Node 3 Nodemanager AppMaster - 1 Datanode Resourcemanager Nodemanager Datanode Namenode Nodemanager Driver Executor Executor Container - 2 Container - 3 Datanode HDFS Block 1 HDFS Block 2 HDFS Block 3 #VIRT1445BU CONFIDENTIAL 9

10 Introduction Previous VMware tests running MapReduce v1 apps show virtualized Hadoop performance at parity or faster than native VMworld 2017 Content: Not for publication Last year: saw same conclusion using newer Spark and MapReduce v2 applications running on YARN, in a highly available cluster typical of real world customer configurations #VIRT1445BU CONFIDENTIAL 10

11 Introduction The tests to be described in this talk updated the previous studies with Better hardware 13 servers with faster processors, more cores, larger memory All flash disks New Spark Machine Learning Library applications Additional virtualized configurations 1, 2 and 4 VMs per host New white paper available: bigdata-vsphere65-perf.pdf VMworld 2017 Content: Not for publication #VIRT1445BU CONFIDENTIAL 11

12 Test Configurations

13 Big Data Cluster Each server: 2x Intel Xeon E v GHz, 16 cores 512 GB Memory 2x 1.2 TB HDD 4x 800 GB NVMe 12x 800 GB SSD VMworld GbE Ethernet Switch Content: Not for publication 10 GbE Ethernet Switch 13 Hewlett Packard Enterprise DL380 Gen 9 Servers #VIRT1445BU CONFIDENTIAL 13

14 Server Configuration Component Quantity/Type Server HPE DL 380 Gen 9 Processor Logical Processors (incl. hyperthreads) 64 Memory NICs Hard Disk Drives Non-Volatile Memory Express storage Solid State Disks RAID Controller Remote Access 2x Intel Xeon CPU E GHz w/16 cores each 512 GiB (16x 32 GiB DIMMs) 2x 1 GbE ports + 4 x 10GbE ports 2x 1.2TB 12G SAS 10K 2.5in HDD RAID 1 for OS 4x 800GB NVMe PCIe NodeManager traffic 12x 800GB 12G SAS SSD DataNode traffic HPE Smart Array P840ar/2G Controller HPE ilo Advanced #VIRT1445BU CONFIDENTIAL 14

15 Flash Disks Non-Volatile Memory Express storage Low latency solid state disk storage Attaches directly to PCI bus 4 per server Used for NodeManager traffic (high R/W I/O) Solid State Disks Low latency storage Controlled by HPE Smart Array RAID controllers 12 per server Used for DataNode traffic (large sequential R/W) #VIRT1445BU CONFIDENTIAL 15

16 Virtualized Cluster #VIRT1445BU CONFIDENTIAL 16

17 Bare Metal Cluster #VIRT1445BU CONFIDENTIAL 17

18 Worker Node Configuration Bare Metal vs. Virtualized Component Bare Metal 1 VM Per Host 2 VMs Per Host 4 VMs Per Host Virtual CPUs Memory 512 GiB 480 GiB 240 GiB 120 GiB Container Memory 448 GiB 432 GiB 208 GiB 104 GiB Container vcores NodeManager Drives 4x 740 GB NVMe 4x 740 GB NVMe 2x 740 GB NVMe 1x 740 GB NVMe DataNode Drives 12x 741 GB SSD 12x 741 GB SSD 6x 741 GB SSD 3x 741 GB SSD #VIRT1445BU CONFIDENTIAL 18

19 Total Cluster YARN Resources Component Bare Metal 1 VM Per Host 2 VMs Per Host 4 VMs Per Host YARN container memory per VM or bare metal server YARN container vcores per VM or bare metal server Number of VMs or servers per cluster 448 GiB 432 GiB 208 GiB 104 GiB YARN container memory per cluster 4480 GiB 4320 GiB 4160 GiB 4160 GiB YARN container vcores per cluster #VIRT1445BU CONFIDENTIAL 19

20 Hadoop/Spark Role Assignments Node Gateway Master1 Master2 Workers Roles Cloudera Manager, ZooKeeper Server, HDFS JournalNode, HDFS gateway, YARN gateway, Hive gateway, Spark gateway HDFS NameNode (Active), YARN ResourceManager(Standby), ZooKeeper Server, HDFS JournalNode, HDFS Balancer, HDFS FailoverController, HDFS HttpFS, HDFS NFS gateway HDFS NameNode (Standby), YARN ResourceManager (Active), ZooKeeper Server, HDFS JournalNode, HDFS FailoverController, YARN JobHistory Server, Hive Metastore Server, Hive HDFS DataNode, YARN NodeManager, Spark Executor #VIRT1445BU CONFIDENTIAL 20

21 Software Components Used in Test Component Version vsphere 6.5.0, Guest Operating System Centos 7.3 Cloudera Distribution of Hadoop Cloudera Manager Hadoop, HDFS, YARN, MapReduce2 Spark Hive ZooKeeper Java MySQL cdh cdh cdh cdh Oracle 1.8.0_111-b Community Server #VIRT1445BU CONFIDENTIAL 21

22 Workloads

23 Workloads MapReduce TeraSort Suite Most popular Hadoop test, supplied with distribution, exercises CPU, memory, disk, network TeraGen generates specified number of 100 byte records 1, 3, and 10 TB used in tests TeraSort sorts TeraGen output TeraValidate validates TeraSort output is in sorted order NOTE: TeraSort in MapReduce2 has changed; results not directly comparable to MapReduce1 TestDFSIO Hadoop Distributed File System (HDFS) stress tool, supplied with distribution Generates specified number of files of a specified size In these tests GB, 3GB and 10GB files were created for total size of 1, 3, and 10 TB #VIRT1445BU CONFIDENTIAL 23

24 Workloads Spark Three standard analytic programs from the Spark MLLib (Machine Learning Library) were driven using spark-perf from Databricks, Inc. ( K-means Clustering Groups input into a specified number, k, of clusters in a multi-dimensional space Used for analytic tasks such as customer segmentation for purposes of ad placement or product recommendations Training datasets from 1 to 3 TB tested Logistic Regression Classification Binary classifier given an input with, say, 20 features, determine if the input falls in a class or not Used in spam filters, credit card fraud detectors Training datasets from 1 to 3 TB tested VMworld 2017 Random Forest Decision Trees Automates any kind of decision making or classification algorithm Runs an ensemble of decision trees to in order to reduce the risk of overfitting the training data Training datasets from 1 to 3 TB tested Content: Not for publication #VIRT1445BU CONFIDENTIAL 24

25 Performance Results

26 TeraSort Suite Performance - 1, 3 and 10 TB #VIRT1445BU CONFIDENTIAL 26

27 Results TeraSort Virtualized TeraGen faster than bare metal due to smaller number of disks per DataNode Virtualized TeraSort (4 VMs per host) faster than bare metal due to benefits of NUMA (nonuniform memory access) locality, except for 10TB case, where extra memory in bare metal prevails Virtualized TeraValidate about same as bare metal (mainly reads) Within virtualized platforms 4 VMs per host is fastest, followed by 2, then 1 due to optimum number of disks per DataNode Excellent (linear) scaling from 1 to 3 to 10TB #VIRT1445BU CONFIDENTIAL 27

28 TestDFSIO Performance 1, 3 and 10 TB #VIRT1445BU CONFIDENTIAL 28

29 Results TestDFSIO Virtualized TestDFSIO (4 VMs per host) significantly faster than bare metal due to benefits of NUMA locality, smaller number of disks per DataNode 47.5 GiB/s maximum cluster disk I/O vs for bare metal Excellent (linear) scaling from 1 to 3 to 10TB Within virtualized platforms 4 VMs per host is fastest, followed by 2, then 1, due to optimum number of disks per DataNode #VIRT1445BU CONFIDENTIAL 29

30 Spark K-means Performance #VIRT1445BU CONFIDENTIAL 30

31 Spark Logistic Regression Performance #VIRT1445BU CONFIDENTIAL 31

32 Spark Random Forest Performance #VIRT1445BU CONFIDENTIAL 32

33 Results Spark Datasets ran in memory, Spark code was NUMA-aware Thus virtualized advantage was minimized but 4 VMs per host was still faster due to faster transfer of data within host than through network All workloads showed linear scaling as dataset size increased or distribution #VIRT1445BU CONFIDENTIAL 33

34 Best Practices

35 Best Practices Hardware Selection Memory, CPU increasingly critical for newer technologies like Spark CPU: larger core count equally as important as faster clock speed Use flash disks appropriately Networking 10GbE crucial, starting to see 25 GbE Number of servers determined by size of workload, number of concurrent users #VIRT1445BU CONFIDENTIAL 35

36 Best Practices Software Selection Hadoop Distribution Open source Apache Hadoop is available but most production Hadoop users employ a distribution such as Cloudera, Hortonworks or MapR which provides deployment and management tools, performance monitoring, and support Operating System Each distribution supports a range of Linux operating systems including RedHat/CentOS 6 and 7, SUSE Linux Enterprise Server 11 and 12, and Ubuntu 12 and 14. Java JDK 1.7 and 1.8 Database (for management and Hive Metastore) MySQL, PostgreSQL, Oracle Check distribution for details #VIRT1445BU CONFIDENTIAL 36

37 Best Practices vsphere NUMA Configuration NUMA (non-uniform memory access): A processor s access to its local memory is faster than to memory on other processors Processor Cache Memory Processor Cache Memory #VIRT1445BU CONFIDENTIAL 37

38 Best Practices vsphere NUMA Configuration Create 2 or more VMs on a 2-processor server to optimize NUMA locality VM Processor Cache VM Processor Cache Memory Memory #VIRT1445BU CONFIDENTIAL 38

39 Best Practices vsphere Configuration Reserve about 5-6% of total server memory for ESXi, use remainder for VMs Limit number of disks per DataNode to maximize utilization of each disk 4 to 6 is a good starting point Use Eager Zeroed Thick format for virtual machine disks (VMDKs), use ext4 or xfs filesystem in guest OS Use VMware paravirtual SCSI (pvscsi) adapter for disk controllers; use all 4 virtual SCSI controllers available in vsphere 6.5 Use vmxnet3 network driver; configure virtual switches with MTU=9000 for jumbo frames #VIRT1445BU CONFIDENTIAL 39

40 Tuning

41 Tuning: Operating System Parameters Turn down aggressiveness of memory swapping Set vm.swappiness = 0 in /etc/sysctl.conf Disable transparent hugepage compaction echo never > /sys/kernel/mm/transparent_hugepage/defrag Enable jumbo frames on network Add MTU=9000 to /etc/sysconfig/network-scripts/ifcfg-e, configure on physical and virtual switches #VIRT1445BU CONFIDENTIAL 41

42 Tuning: YARN Cluster Parameters yarn.nodemanager.resource.cpu-vcores and yarn.nodemanager.resource.memory-mb Tells YARN how many resources it has for containers for tasks/executors A vcore is a YARN virtual core Can be set 1x - 4x number of physical cores Set to 2x number of physical cores in these tests = number of hyperthreads (bare metal) = 64 = number of vcpus (virtualized) =16 (with 4 VMs per host) Container memory: server/vm memory - operating system requirements DataNode/NodeManager JVM heap Bare Metal: 512 GiB on server => 448 GiB container memory Virtualized: 512 GiB on server 32 GiB for ESX = 480 GiB 4 VMs per host: 480/4 = 120 GiB => 104 GiB container memory #VIRT1445BU CONFIDENTIAL 42

43 Tuning: Hadoop Job Parameters dfs.blocksize tradeoff between size and number of tasks 256 MB good initial choice for most workloads Set mapreduce.task.io.sort.mb larger than dfs.blocksize to minimize spills to disk eg. 400 MB dfs.replication 3 typical for availability mapreduce.{map reduce}.memory.mb and mapreduce.{map reduce}.cpu.vcores Memory and vcores to be allocated by YARN for containers to run map and reduce tasks Can specify, otherwise YARN will allocate based on other YARN parameters mapreduce.job.{maps reduces} VMworld 2017 Content: Not for Set as needed to override YARN calculation of number of tasks Remember that map and reduce tasks normally overlap for part of a job publication #VIRT1445BU CONFIDENTIAL 43

44 Tuning: Spark on YARN spark.executor.cores, spark.executor.memory Play same role for Spark executors do as map/reduce task memory and vcore assignment do for Map Reduce spark.yarn.executor.memoryoverhead Set if default (10% of spark.executor.memory) is insufficient VMworld 2017 Content: Not for publication #VIRT1445BU CONFIDENTIAL 44

45 Machine Learning An Overview

46 #VIRT1445BU CONFIDENTIAL 46

47 What is Machine Learning? Training Data (Big) Samples from History with Labels training New Sample Transaction Data testing Mathematical Model Mathematical Model Mathematical Model Machine Learning algorithms try to make predictions based on training data that is given to a mathematical model (e.g. a linear regression algorithm) VMworld 2017 Find the minimum difference between the model s prediction and the already known outcomes in the labels (i.e. minimize the loss function ) Spark is a foundational technology for this type of application Classification or Prediction Content: Not for publication #VIRT1445BU CONFIDENTIAL 47

48 Example: A Linear Classifier f (x i, W, b) = Wx i + b x: Example data W: weights b: bias Source: Stanford University class cs231n #VIRT1445BU CONFIDENTIAL 48

49 What Have We Seen so Far? Performance results show that virtualized Spark and Hadoop is 10% better than native Even better results with All Flash storage than with traditional disks seen last year Four virtual machines per server is the sweet spot Contemporary workloads such as Machine Learning perform very well on vsphere #VIRT1445BU CONFIDENTIAL 49

50 Summary Each aspect of the stack should be examined using our guidelines for tuning opportunities Powerful new technologies like YARN, Spark and Machine Learning apps yield excellent performance on vsphere when tuned properly Correctly configured virtualized Hadoop clusters on vsphere outperformed bare metal on all Spark workloads Production requirements can be met without sacrificing performance on virtualized environments Big Data on vsphere is ready for production environments For details see bigdata-vsphere65-perf.pdf VMworld 2017 Content: Not for publication #VIRT1445BU CONFIDENTIAL 50

51 Introducing vsphere Scale-Out for Big Data and HPC Workloads New package that provides all the core features required for scale-out workloads at an attractive price point Features Packaging Hypervisor, vmotion, vshield Endpoint, Storage vmotion, Storage APIs, Distributed Switch, I/O Controls & SR- IOV, Host Profiles / Auto Deploy and more Sold in Packs of 8 CPU at a cost-effective price point Licensing EULA enforced for use w/ Big Data/HPC workloads only 51

52 References 1. Big Data Performance on vsphere Virtualized Hadoop Performance with VMware vsphere 6 on High Performance Servers 3. Virtualized Hadoop Performance with VMware vsphere Benchmarking Case Study of Virtualized Hadoop Performance on vsphere Hadoop Virtualization Extensions (HVE) Virtualization-Extensions-on-VMware-vSphere-5.pdf VMworld 2017 Content: Not for publication #VIRT1445BU CONFIDENTIAL 52

53 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs #VIRT1445BU CONFIDENTIAL 53

54 Extreme Performance Series Barcelona SER2724BE Performance Best Practices SER2343BE vsphere Compute & Memory Schedulers SER1504BE vcenter Performance Deep Dive SER2849BE Predictive DRS Performance & Best Practices VIRT1445BE Fast Virtualized Hadoop and Spark on All-Flash Disks VMworld 2017 VIRT1397BE Optimize & Increase Performance Using VMware NSX VIRT1052BE Monster VM Database Performance FUT2020BE Wringing Max Perf from vsphere for Extremely Demanding Workloads Content: Not for publication #VIRT1445BU CONFIDENTIAL 54

55 Extreme Performance Series - Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. #VIRT1445BU CONFIDENTIAL 55

56 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! VMworld 2017 Content: Not for publication #VIRT1445BU CONFIDENTIAL 56

57

58