Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect

Transcription

1 Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect

2 Igor Roiter Big Data Cloud Solution Architect Working as a Data Specialist for the last 11 years 9 of them as a Consultant specializing in Ad-tech, Fin-tech and the Gaming industries Certified NoSQL & BigData Cloud Engineer IoT enthusiast in my spare time

3 Building a Big Data Application on AWS Ingest Store Analyze AWS Database Migration Service AWS Snowball S3 Glacier EC2 EMR Lambda Kinesis AWS IoT DynamoDB Elasticsearch Service QuickSight Redshift Athena AWS Database Migration Service AWS Data Pipeline

4 Building a Big Data Application on AWS How to get corporate data into cloud Application hosts Corporate DBMS AWS Cloud

5 Building a Big Data Application on AWS Getting data into AWS Cloud Application hosts Corporate DBMS AWS Database Migration Service AWS Cloud

6 AWS Database Migration Service AWS Database Migration Service Simple usage No data-loss, No downtime Low cost only pay for compute resources

7 Building a Big Data Application on AWS Where do we migrate the data to Application hosts Corporate DBMS AWS Database Migration Service AWS Cloud

8 Building a Big Data Application on AWS Data Warehouse Application hosts Corporate DBMS AWS Database Migration Service AWS Cloud Redshift

9 AWS Redshift Structured Data Processing Redshift Fully Managed columnar data warehouse Standard SQL support Petabyte-scale Fault-tolerant

10 Building a Big Data Application on AWS How about some BI Application hosts Corporate DBMS AWS Database Migration Service AWS Cloud Redshift

11 Building a Big Data Application on AWS BI tool QuickSight Application hosts Corporate DBMS AWS Database Migration Service AWS Cloud Redshift

12 QuickSight QuickSight Managed BI tool Scales to 100s of users Auto-suggest the best visualizations for your data 1/10th the cost of other popular BI software

13 What About Unstructured Data? What if your data is unstructured? What if you don t need all the raw data? What if you need to combine multiple data sets?

14 Building a Big Data Application on AWS Handling unstructured data QuickSight Application hosts Corporate DBMS AWS Database Migration Service AWS Cloud Redshift

15 Building a Big Data Application on AWS AWS Lambda Event driven data transformation QuickSight Application hosts Unstructured Raw data in S3 Lambda trigger AWS Lambda Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

16 AWS Simple Storage Service (S3) S3 Managed object storage No administration No capacity limit Data resilience $0.02 per GB/month

17 AWS Lambda Serverless Event Processing AWS Lambda Function as a service Write code in NodeJS, Python or Java Event driven Low cost

18 Need More Than a Lambda? What can we use when further throughput is required?

19 Building a Big Data Application on AWS Need more throughput QuickSight Application hosts Unstructured Raw data in S3 Lambda trigger AWS Lambda Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

20 Building a Big Data Application on AWS Unstructured data transformation using EMR QuickSight Application hosts Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

21 EMR - Unstructured Data Processing EMR Fully managed, cloud pre-tuned Hadoop eco-system Hadoop 2.7.3, Hive 2.3.1, Spark 2.2.0, Hbase Hbase on S3, Presto, Flink On-demand and Spot Instances Fully integrated with S3 Provision cluster for a job then terminate

22 Semi-structured / Unstructured data What about ad-hoc queries for exploring new data?

23 Building a Big Data Application on AWS Ad-hoc query on Raw data QuickSight Application hosts Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

24 Building a Big Data Application on AWS Ad-hoc query service on S3 buckets Athena QuickSight Application hosts Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

25 AWS Athena Serverless Query Processing Athena Serverless query service for querying data in S3 No data load/etl, data is queried on S3 Pay-per-query based on scanned data amount Standard SQL

26 Building a Big Data Application on AWS Pre-process data to columnar data format EMR Parquet (columnar) Data In S3 Athena QuickSight Application hosts Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

27 All Batch - No Fun What about some real-time processing?

28 Building a Big Data Application on AWS No more batch EMR Parquet (columnar) Data In S3 Athena QuickSight Application hosts Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

29 Building a Big Data Application on AWS Add a real-time layer kinesis streams Application hosts Kinesis Streams KCL EMR Parquet (columnar) Data In S3 Athena QuickSight Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

30 Kinesis Streams Kinesis Streams Managed Real-time stream processing Dynamically adjust throughput of the stream Data resilience Produce data into stream using KPL, Read it with KCL

31 Kinesis Firehose Kinesis Firehose Load streaming data into AWS data stores: S3, RedShift Fully managed, auto-scaled Integrated Lambda

32 Building a Big Data Application on AWS Add a real-time layer kinesis streams Application hosts Kinesis Streams KCL EMR Parquet (columnar) Data In S3 Athena QuickSight Unstructured Raw data in S3 EMR Structured Data In S3 Corporate DBMS AWS Database Migration Service Redshift

33 Summary Big Data on AWS Pay for what you use only Avoid over/under provision Dynamic scale everything Go to prod faster then ever

34 Reinvent 2017 So what's new?

35 S3/Glacier Select Preview S3 Select Pulls out only the data you need from an object Offloads filter processing from application to S3 service S3/Glacier Select SDK supports Java & Python Glacier Select

36 Kinesis Video Streams Kinesis video streams Fully managed video ingestion and storage service Secure SDKs for devices to stream video to AWS APIs for access and retrieve indexed video fragments based on tags and timestamps

37 AWS Lambda updates AWS Lambda Added ability to shift traffic between 2 AWS Lambda versions based on pre-assigned weights Doubled available memory for a function from 1536MB to 3008MB Added ability to add a concurrency limit on a Lambda functions The AWS Lambda console has been updated with enhancements: Cloud-9 based editor, Improved monitoring, and more

38 AWS IoT Analytics Preview AWS IoT Analytics Built-in IoT Analytics SQL query engine Stores the processed device data in a time-series data Scales automatically to support up to petabytes of IoT data Apply machine learning to your IoT data with hosted Jupyter notebooks, right from the IoT console

39 Get in touch at Thank You!