Best Practices and Performance Tuning on Amazon Elastic MapReduce

Size: px

Start display at page:

Download "Best Practices and Performance Tuning on Amazon Elastic MapReduce"

Gerald Bradley
5 years ago
Views:

1 Best Practices and Performance Tuning on Amazon Elastic MapReduce Michael Hanisch Solutions Architect Amo Abeyaratne Big Data and Analytics Consultant ANZ , Amazon Web Services, Inc. or its Affiliates. All rights reserved.

2 Challenge: Data is Everywhere - Sensors - Devices - Logs - Operations -

3 Size: Growing in Petabytes If every byte was a word, and you take a second to read a word, it will take you 32 million years to read a whole Petabyte Migrate & Collect Store &Transform Process & Analyze Visualize & Predict

4 How do we deal with it?

5 Strategy: Divide and Conquer Hadoop Hadoop on EC2 Amazon EMR Hadoop has to scale? Managing on your own can be a LOT of work A Managed Hadoop Framework in the Cloud

6 Tool: Amazon EMR

7 Agenda Challenge: Data is Everywhere Size: Growing in PBs Strategy: Divide & Conquer Tool: Amazon EMR Why EMR? Well Architected EMR Design for Production

8 Why EMR?

9 Why EMR? Automated Decoupled Elastic Integrated Current Low-cost

10 Why EMR? : Automation EC2 Provisioning Cluster Setup Hadoop Configuration Monitoring and Failure Handling Job submission Installing Applications

Amazon S3 Save with spot and reserved instances

11 Why EMR? : Decoupled Architecture Separate compute and storage Resize and shutdown with no data loss Point multiple clusters at the same data on Amazon S3 Save with spot and reserved instances HDFS for iterative and disk I/O intensive workloads Easily evolve infrastructure as technology evolves

12 Why EMR?: Elastic Core nodes for scaling HDFS Task nodes for scaling processing power Intelligent resize: Wait for work to finish before stopping instances Use instance groups: to manage different instance types in the same cluster

Spark 1.5 September 9, 2015 September 2015 Spark 1.5.2 November 9, 2015 November 2015 Spark 1.

13 Why EMR?: Current Application Open source release EMR release Spark 1.5 September 9, 2015 September 2015 Spark November 9, 2015 November 2015 Spark 1.6 January 4, 2016 January 2016 Spark March 9, 2016 April 2016

14 Why EMR?: Integration with other services Amazon KMS Amazon Kinesis Amazon Redshift Amazon S3 Amazon DynamoDB IAM for Authentication AWS Data Pipeline

15 Why EMR?: Low-cost Spot instances: Bid for unused EC2s at up to 90% less price Reserved instances: For persistent clusters, make use of EC2 reserved instances to save up to 50% Transient clusters: Terminate the cluster when not in use

16 Agenda Challenge: Data is Everywhere Size: Growing in PBs Strategy: Divide & Conquer Tool: Amazon EMR Why EMR? -Automated -Decoupled -Elastic -Integrated -Current -Cost-efficient Well Architected EMR Design for Production

17 Well-Architected Amazon EMR

18 Well Architected EMR: Design for Production Performance Reliability Security Cost efficiency

19 Performance Efficiency Choice of Instance Type Choice of Storage Choice of application or framework

20 Performance: Choice of instance type - Master Less than 50 nodes? No Heavy network I/O M3.2xlarge or larger YES YES M3.xlarge C3 family or R3 with Enhanced networking

21 Performance: Choice of instance type - Workers General Compute Memory Storage M3 Family M1 Family C1 Family C3 Family CC1.4xlarge CC2.8xlarge M2 Family R3 Family Cr1.8xlarge D2 Family I2 Family Batch Process Machine Learning Interactive Analysis Large HDFS

22 How many nodes do I need?

23 Performance: Sizing - How many tasks will you have to execute? - Based on data size, number of files/splits - Based on the job you are evaluating - When do you need to get done? - How much parallelism do you need? - How many tasks can each node process in parallel?

24 Performance: Sizing - Which other jobs are you running? - How much data do you need to store locally? - Which files are reused 3x or more?

25 Performance: Sizing Guidelines: - Size based on HDFS storage first if needed - Add enough (task) nodes to handle processing - Do not add more than 5 tasks nodes per core node - Prefer smaller clusters of larger machines

26 Performance: Tune it e.g. for Spark Enable Dynamic Allocation of executors Executor Memory Executor Cores YARN containers Driver Memory Deployment mode on YARN (Cluster Client?)

Performance: Choice of storage EMRFS (S3) HDFS - Instance storage - Amazon EBS - Ability to decouple - Reliable and durable - Cost efficient - Works well for jobs that read a dataset once per run.

27 Performance: Choice of storage EMRFS (S3) HDFS - Instance storage - Amazon EBS - Ability to decouple - Reliable and durable - Cost efficient - Works well for jobs that read a dataset once per run. - Need a persistent cluster - Reliability is configurable. But need multiple nodes to achieve replication factor. - Great for jobs with iterative reads on the same dataset like machine learning Combine with S3DistCp and move data from S3 once to HDFS for iterative workloads

28 Performance: S3 vs HDFS at Netflix

29 S3: Range GET Data locality doesn t matter? EMR worker nodes GET Range 0-64MB GET Range MB GET Range MB GET Range (n-64)-nmb S3 object (use larger files)

30 S3: Avoid sequential key names Amo/ var/ Server/ B S V W UTF-8 Binary ordering 100/s 100/s 100/s S3 partition S3 partition S3 partition

32 S3: Avoid sequential key names

33 S3: Real world heavy EMRFS users 1.2TB/Day logs 30TB /Day data 250 Hadoop Jobs 75Billion transactions/day 5 Petabytes of Data 10+PB Data Warehouse on Amazon S3 > 1PB read each day

34 Performance: Compression Always compress data on S3 to reduce bandwidth & cost.

35 Performance: Choice of Framework - Embarrassingly parallel? - Can be optimized with a DAG? Count these words A B Count = 1 These = 1 Words = 1 C D Count = 1 These = 1 Words = 1 E

cluster will give you HA - Source of truth for data - Store config &

36 Reliability Externalize Hive Metastore Data and Applications on S3 Automate - Store your metadata outside the cluster on RDS - Multi-AZ RDS cluster will give you HA - Source of truth for data - Store config & applications on S3, too - Consistent View - Bootstraps - Config options - Cloudformation

37 Reliability: Hive Metastore on RDS MySQL [ { "Classification": "hive-site", "Properties": { "javax.jdo.option.connectionurl": "jdbc:mysql:\/\/emr-hive- metastore.cauttwbz9zri.us-east- 1.rds.amazonaws.com:3306\/hive?createDatabaseIfNotExist=true", ] "javax.jdo.option.connectionusername": "admin", "javax.jdo.option.connectionpassword": "Passw0rd!" }, "configurations" : [] } Config file lives on s3://<bucketname>/hive-meta-config.json

38 Security Encryption IAM Roles VPC - Server side - Client side - HDFS Transparent - RPC with SSL - File system with LUKS - Secure Integration with AWS services - Private Subnets - S3 endpoints - NAT

39 Security Encryption - Server side - Client side - HDFS Transparent - RPC with SSL - File system with LUKS S3 Server-side encryption (SSE): - Using S3-managed keys or - Using KMS-managed keys (SSE-KMS) - Configured at cluster start time

40 Security: Server-Side Encryption (SSE-KMS) AWS KMS 0utput writes via EMRFS with Client-side Encryption enabled Amazon S3 Bucket EMRFS with Client-side Encryption Amazon S3 Bucket

41 Security Encryption - Server side - Client side - HDFS Transparent - RPC with SSL - File system with LUKS S3 Server-side encryption (SSE-KMS): 1. Define Customer Master Key in KMS 2. Create key policy that allows EMR s instance profile / role to use this key 3. Configure cluster with CMK KeyID or Alias

42 Security: Sample Key Policy { "Sid": "Allow use of the key", } [ ] "Effect": "Allow", "Principal": { "AWS": [ }, "Action": [ "arn:aws:iam::xxxxxxxxxxxx:role/emr_defaultrole"] "kms:encrypt", "kms:decrypt", "kms:reencrypt*", "kms:generatedatakey*", "kms:describekey" ], "Resource": "*, "Condition": { "ForAnyValue:StringLike": { "kms:encryptioncontext:aws:s3:arn": [ "arn:aws:s3:::bucket1/*", "arn:aws:s3:::bucket2/*" ] } }

43 Security: Enable SSE-KMS with selected CMK [ { "Classification":"emrfs-site", "Properties": { "fs.s3.enableserversideencryption": "true", "fs.s3.serversideencryption.kms.keyid":"a4567b ab a " } } ]

44 Security Encryption - Server side - Client side - HDFS Transparent - RPC with SSL - File system with LUKS Client-side encryption: - Using custom key materials provider - Configured at cluster start

Security: End to End Encryption Amazon S3 Bucket AWS S3 SDK

Client-side Encryption enabled Encrypted Object spark.ssl.

45 Security: End to End Encryption Amazon S3 Bucket AWS S3 SDK AmazonS3EncryptionClient() AWS KMS 0utput writes via EMRFS with Client-side Encryption enabled Encrypted Object spark.ssl.enabled hadoop.rpc.protecti on hadoop.ssl.enabl ed mapreduce.shuffle.ssl.enabled Amazon S3 Bucket EMRFS with Client-side Encryption HDFS transparent encryption with Hadoop KMS LUKS with bootstrap action for local file systems

46 Cost Efficiency Using cost-effective resources S3 instead of HDFS for larger datasets? Taking advantage of Spot and Reserved instances? Optimize over time Monitor and watch out for new instance types, features that may reduce cost. Matching Supply and Demand Is the cluster big enough? Can we make it transient? Monitor the usage with Ganglia and Amazon CloudWatch alarms Spot instances: Bid for unused EC2s at up to 90% less price Reserved instances: For persistent clusters, make use of EC2 reserved instances to save up to 50% Transient clusters: Terminate the cluster when not in use

47 Agenda Challenge: Data is Everywhere Size: Growing in PBs Strategy: Divide & Conquer Tool: Amazon EMR Why EMR? -Automated -Decoupled -Elastic -Integrated -Current -Cost-efficient Well Architected EMR Design for Production -Performance tuning -Reliability measures -Security facts -Cost efficiency

48 Thank you!

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Dr. Steffen Hausmann, Solutions Architect Michael Hanisch, Manager Solutions Architecture November 18 th, 2016 Stream Processing Challenges