Best Practices for Deploying Hadoop Workloads on HCI Powered by vsan

Transcription

1 Best Practices for Deploying Hadoop Workloads on HCI Powered by vsan Chen Wei, ware, Inc. Paudie ORiordan, ware, Inc. #vmworld HCI2038BU #HCI2038BU

2 Disclaimer This presentation may contain product features or functionality that are currently under development. This overview of new technology represents no commitment from ware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new features/functionality/ technology discussed or presented, have not been determined. 2

3 Agenda Hadoop on HCI What is means to run Hadoop on vsan Hadoop deployment considerations Configuration, Roles, Sizing and Scale Out What Have We Done Configuration layout/test Cases/Test Results What Did We Learn Performance, Availability, Operation and Tradeoffs 3

4 HCI Integrates Servers and Storage Traditional 3-Tiered Architecture Complex and Separate Silos Virtualization Servers and Blades External Storage Networking Hardware Hyper-Converged Infrastructure Unified Management Virtualization Compute Storage Network Server + Storage Network Built on Industry-Standard Servers and Switches Lower total costs Greater agility and scale Simplified management 4

5 What is vsan Disk Group Disk Group ware vsphere & ware vsan On each host with local disks we will create a disk group(s) Each disk group has cache and capacity A host can have multiple disk groups, these contribute capacity to a single shared datastore The vsan Datastore is accessible by all hosts in the cluster automatically Disk Group Disk Group Disk Group Disk Group 5

6 vsan Objects Snapshot Home Swap DK Delta Disk Memory Delta Home Namespace (X, NVRAM, etc.) Swap (virtual memory swap) Virtual Disk (DK) Snapshot (delta) Snapshot (delta) 6

7 vsan Datastore Distributed scale out storage FD1 FD2 FD3 FD4 RAID-1 FTT=1 replica replica witness Rack Rack Rack Rack vsan uses multiple storage components on commodity servers and storage to implement a distributed Datastore across the ESXi hosts in a vsphere cluster You can specify explicit fault domains to increase availability Protect against rack failure. Ensures other copy/replica does NOT live in the same rack as first copy 7

8 Storage Policy Based Management Storage Policy Based Management framework used by vsan and VVol SPBM Framework vsphere vsan Storage Policies Define storage related settings for protection and performance FTT for Protection Striping for performance Apply per, or even per DK level to meet business goals Key to software defined storage (SDS) architecture and management at scale 8

9 What is Hadoop? Supports the processing of large data sets in a distributed computing environment. Built on commodity hardware Scale out architecture Key Technologies Map Reduce Hadoop Distributed File System 9

10 Hadoop HDFS filesystem Distributed scale out filesystem Name Node Resource Manager Node Manager Data Node Node Manager Data Node Switch Node Manager Data Node Node Manager Data Node Node Manager Data Node World Switch Node Manager Data Node Node Manager Data Node Node Manager Data Node Name Node Resource Manager Node Data Manager Node Data Node Node Manager Data Node Rack 1 Rack 2 Rack 3 Rack 4 Implements a distributed file system across commodity storage and servers that provides high-performance access to data across highly scalable Hadoop clusters. You can create explicit rack id s to increase availability Protect against rack failure. Ensures other copy/replica does NOT live in the same rack as first copy 10

11 Hadoop s and Roles Gateway Master s Worker s Cloudera Manager Zookeeper Server HDFS Journal Node HDFS Gateway YARN Gateway Hive Gateway Spark Gateway HDFS Name Node (Active/Standby) YARN Resource Manager(Standby/Active) Zookeeper Server HDFS Journal Node HDFS Balancer HDFS Failover Controller HDFS HTTP FS HDFS NFS Gateway HDFS Data Node YARN Node Manager Spark Executor Impala Daemon 11

12 Host Failure with vsan and HA vsan Policy FTT=1 vsan Policy Host Failure to Tolerate = 1 vsphere HA Enabled SSD SSD SSD Component SSD SSD SSD Component SSD SSD SSD Component SSD SSD SSD vsan Objects stay available after component failure Capacity Disk / Cache Disk / Host Failure vsphere HA ensures s restart on different hosts incase of failure Failure tolerance independent of HDFS replication factor 12

13 Host Affinity for Next Generation Applications (RPQ only) Accommodate net-gen apps optimized for shared-nothing architectures Hadoop Data Node vsphere Hadoop Data Node vsan Datastore vsan Policy ensures data resides on the same host runs For next-gen apps that provide built-in app resiliency Hadoop Splunk Cassandra s run with FTT=0 for space efficiency Application settings determines availability 13

14 Host Affinity and HDFS Replication Factor FTT=0 with Host Affinity vsan Policy FTT=0 HDFS replication factor = 3 SSD SSD SSD object object SSD SSD SSD SSD SSD SSD object object object object SSD SSD SSD HDFS block rebuild No data loss, some performance degradation Capacity or Cache Disk Failure = Host Failure s unavailable with failed host Host failure tolerance depends on HDFS redundancy factor HDFS to rebuild the blocks on the other s object object 14

15 Hadoop Deployment Considerations Configuration, roles, sizing and scale out 15

16 Recommended vsan Starter Configuration Proof of Concept stage All Flash with 6(+) * Hosts 2(+) * Disk Groups per Host 2(+) * 10(+) GbE Ports per Host vsan 6.7(+) Disable Deduplication/Compression Disable Checksum Cache Capacity SSD SSD SSD Ne SSD SSD SSD Disk Group 1 SSD SSD SSD Ne SSD SSD SSD Disk Group 2 16

17 Hadoop on vsan Deployment Layout Single Small Cluster with FTT=1, HA and DRS enabled Infrastructure Nodes Master 1 Master 2 Host 1 Host 2 Gateway Cloudera Manager Data Nodes DataNode 1 DataNode 11 Host 3 Host 8 DataNode 2 DataNode 12 Separate Roles into Infrastructure and Compute Dedicate two or more hosts for Hadoop infrastructure Virtual Machines HVE (Hadoop Virtualization Extensions) to prevent the same replicas on two DataNode s on the same host Set up HA/DRS Rules to ensure s don t land on same physical host 17

18 Hadoop on vsan Deployment Layout Two Cluster Design Infrastructure Nodes. FTT=1 HA and DRS enabled. Master 1 Master 2 Host 1 Host 4 Gateway Cloudera Manager Data / Worker Nodes. FTT=0 with Host Affinity. DRS and HA disabled. DataNode 1 DataNode 11 Host 1 DataNode 2 DataNode 12 Host [up to 64] Separate Clusters for infra and workload Dedicated infra Cluster for for Hadoop infrastructure Virtual Machines Dedicated Workload Cluster for Data /Worker Node s HVE (Hadoop Virtualization Extensions) to prevent the same replicas on two DataNode s on the same host 18

19 How to design deployment Scale Out and Sizing Cluster Size Number of Servers Dedicated to Master s Number of Servers Dedicated to Worker s Number of Worker s x x x x DataNode Configuration CPU: 0.5 * #_of_physical_cores_per_host RAM: 0.4 * Host RAM OS Disk: 100GB Number of Data Disk: 0.5 * #_of_capacity_drive_per_host Size of Data Disk: No Less Than (3* Host RAM) / (2* #_of_data_disk) ** Expected Performance Scaling ** Cloudera Recommendation 19

20 Joint Work Alive on StorageHub! Search for cloudera storagehub vsan # 20

21 What Have We Done Test configuration, test cases and test results 21

22 Hardware Resource vsan Disk and Network Configuration 8x HPE ProLiant DL380 Gen9 Each Server: CPU: 2x Intel Xeon Processors E GHz, 16 cores/socket RAM: 512 GiB NICs: 2x 1 GbE ports (MGMT Network) 2x 10 GbE ports ( Network) - LAG 2x 10 GbE ports (vsan Network) - LAG Nes: 4x 800 GB Ne PCIe SSDs: 12x 800 GB 12G SAS SSD 22

23 vsan Configuration FTT=0 vs. FTT=1 Feature FTT=0, Host Affinity FTT=1, No Host Affinity Data Locality Host Local None Data DKs per Worker 6x 700GB 6x 350GB vsan Storage Used/Capacity / TB / TB Deduplication/Compression Disabled Disabled Checksum Disabled Disabled Stripe Width Number of Copies per DK 1 2 HDFS Configured Capacity TB TB Default HDFS Replication Factor 3 2 HDFS Max File Size at Default Replication Factor TB TB 23

24 How we sized the Hadoop s configurations and roles SPEC GATEWAY MASTER WORKER Quantity vcpu 16 Memory OS 200 GiB CentOS 7.5 x86_64 OS DK Size 250 GB 100 GB 100 GB Data Disks 1x 100 GB 1x 100GB 6x 350(FTT=1)/700(FTT=0) GB 24

25 Software Component Hadoop on vsan Cluster overview Component Version vsphere Guest OS Cloudera Distribution Hadoop Cloudera Manager Hadoop HDFS YARN Spark ZooKeeper CentOS 7.5 x86_ cdh cdh cdh cdh cdh

26 Testing Scenarios and Cases Performance Testing Against CDH Cluster Test Scenarios: FTT=0 with Host Affinity FTT=1 without Host Affinity Host failure Test Cases: Cloudera Storage Validation Testing Hadoop MapReduce TeraSort Suite 1 TB: 10 billion records 3 TB: 30 billion records TestDFSIO Spark 500GB/1TB K-Means Clustering Logistic Regression Classification Random Forest Decision Trees 26

27 Cloudera Storage Validation Testing Cloudera storage validation testing Deploy Hadoop Cluster Configure Cloudera Toolkit Run Microbenchmark Run HBase or Kudu 27

28 Testing Results TeraSort Suite performance testing Performance advantage, FTT=0 over FTT=1 TeraGen: 54.4% -- 1TB 56.7% -- 3TB TeraSort: 20.6% -- 1TB 23.3% -- 3TB TeraValidate: 0.4% -- 1TB 40.3% -- 3TB 28

29 Testing Results TestDFSIO performance testing Performance advantage, FTT=0 over FTT=1 TestDFSIO 41.8% -- 1TB 51.5% -- 3TB 71.2% -- 10TB 29

30 Testing Results Spark performance Performance advantage, FTT=0 over FTT=1 K-Means 5.2% GB 5.4% -- 1TB Logistic Regression 1.8% GB 0.2% -- 1TB Random Forest -1.2% GB -3.4% -- 1TB 30

31 Failover Scenario FTT=0 with Host Affinity vsan Policy FTT=0 with Host Affinity HDFS replication factor = 3 SSD SSD SSD object object SSD SSD SSD SSD SSD SSD object object object object SSD SSD SSD HDFS block rebuild object object No data loss, some performance degradation Capacity or Cache Disk Failure = Host Failure s gone with failed host Host failure tolerance depends on HDFS redundancy factor 31

32 Failover Scenario FTT=1 without Host Affinity vsan Policy FTT=1 HDFS replication factor = 2 vsphere HA Enabled SSD SSD SSD Component SSD SSD SSD Component SSD SSD SSD Component SSD SSD SSD No data loss, some performance degradation Capacity/Cache disk failure has no impact to the Hadoop cluster s protected by HA/DRS Host failure tolerance depends on vsan FTT value 32

33 What Did We Learn Considerations about FTT=0 with Host Affinity and FTT=1 33

34 Take-Aways of Comparing FTT=1 vs. FTT=0 Capacity consideration FTT=1 without Host Affinity Number of copies per DK 2 1 HDFS raw capacity 1x 2x HDFS redundancy factor 2 3 HDFS max file size 3x 4x FTT=0 with Host Affinity 36

35 Take-Aways of Comparing FTT=1 vs. FTT=0 I/O performance and availability FTT=0 with Host Affinity I/O Performance Better than FTT=1 in most cases One copy per DK stays local Little data through vsan network Availability Three copies per HDFS block * One copy per DK => Tolerate up to two hosts failure FTT=1 without Host Affinity I/O Performance Good, passed Cloudera Storage Validation Two copies per DK A lot data transferred through network Availability Two copies perf HDFS block * Two copies per DK => Tolerate one host failure 37

36 Take-Aways of Comparing FTT=1 vs. FTT=0 Failover performance and operational consideration FTT=0 with Host Affinity Single Host Failure Performance impact determined by dataset Need to rebuild the lost DataNode or Master s Disk or Disk Group Failure Same as Host Failure FTT=1 without Host Affinity Single Host Failure Performance impact also determined by data-set Minor performance impact introduced by vsan resync/rebuild process s saved by vsphere HA Single Disk or Disk Group Failure No performance impact Very little impact introduced by vsan resync/rebuild process All the s are up and running 38

37 Take-Aways of Comparing FTT=1 vs. FTT=0 Which to use when you need FTT=0 with Host Affinity Great Performance Good capacity Better availability FTT=1 without Host Affinity Good Performance vmotion, DRS and HA Less impact and easiness of recovering from failover 39

38 Learn About the Future of Hyperconverged Infrastructure at the Innovating Beyond HCI Showcase Keynote #HCI3728KU What is the Digital Foundation? Yanbing Li Senior Vice President & General Manager, Storage and Availability Business Unit, ware John Gilmartin Join Us Tuesday at 11AM to find out: How is ware expanding to capabilities of HCI? Senior Vice President & General Manager, Integrated Systems Business Unit, ware What are the tools for the multi-cloud world? Door prizes will be awarded at each Showcase Keynote don t miss your chance to win! 40

39 Visit the HCI Zone in the Solutions Exchange #vsanfan Experience Solutions Powered by vsan Edge Video Analytics #vsan Core SAP HANA on vsan DataStax #HCIZone Cloud PKS with containerized MySQL Espresso Bar vspeaking Podcast Daily Prizes 41

40 PLEASE FILL OUT YOUR SURVEY. Take a survey and enter a drawing for a ware company store gift card. #vmworld #HCI2038BU

41 THANK YOU! #vmworld #HCI2038BU