Cloud Analytics and Business Intelligence on AWS

Transcription

1 Cloud Analytics and Business Intelligence on AWS

2 Enterprise Applications Virtual Desktops Sharing & Collaboration Platform Services Analytics Hadoop Real-time Streaming Data Machine Learning Data Warehouse Data Pipeline App Services Queuing & Notifications Workflow App streaming Transcoding Search Deployment & Management One-click web app deployment Dev/ops resource management Resource Templates Code Deploy Code Pipeline Code Commit Mobile Services Identity Sync Mobile Analytics Push Notifications Administration & Security Identity Management Access Control Usage & Resource Tracking Service Catalog Key Storage & Management Monitoring and Logs Core Services Compute (VMs, Auto-scaling and Load Balancing) Storage (Object, Block and Archival) CDN Databases (Relational, NoSQL, Caching) Networking (VPC, DX, DNS) Infrastructure Regions Availability Zones Points of Presence

3 Availability 99.99% Durability % A Distributed Object Store Not a file system No Single Points of Failure Eventually consistent Simple Storage Service Highly scalable object storage for the internet 1 byte to 5TB in size % durability Paradigm Performance Redundancy Security Pricing Typical use case Object store Very Fast Across Availability Zones Public Key / Private Key $0.03/GB/month Write once, read many

4 S3 Performance & Scalability Reader Connections Amazon S3 provides near linear scalability 34 secs per terabyte S3 Streaming Performance 100 VMs; 9.6GB/s; $26/hr 350 VMs; 28.7GB/s; $90/hr GB/Second

5 Application Services Deployment & Administration App Services Compute Storage Networking Analytics Databas e AWS Global Infrastructure Amazon Kinesis Managed Service for Real Time Big Data Processing Create Streams to Produce & Consume Data Elastically Add and Remove Shards for Performance Use Kinesis Worker Library to Process Data

6 Amazon Kinesis AWS Endpoint Data Sources Data Sources Availability Zone Availability Zone Availability Zone App.1 [Aggregate & De-Duplicate] App.2 S3 Data Sources Shard 1 Shard 2 Shard N [Metric Extraction] DynamoDB Data Sources App.3 [Sliding Window Analysis] Data Sources App.4 [Machine Learning] Redshift

7 AWS Security Services Cloud HSM Dedicated Tenancy SafeNet Luna SA HSM Device Common Criteria EAL4+, NIST FIPS Deployment & Administration App Services Compute Storage Networking Analytics Databas e AWS Global Infrastructure AWS Key Management Service Implemented on HSM Automated Key Rotation & Auditing Integration with other AWS Services AWS Server Side Encryption AWS Managed Key Infrastructure

8 Structured Data Management

9 Database RDS Redshift Dynamo DB Elasticache Deployment & Administration App Services Compute Storage Networking Analytics Database Relational Database Service Managed Oracle, MySQL & SQL Server Dynamo DB Managed NOSQL Database ElastiCache Managed In Memory Caching Amazon Redshift Massively Parallel Petabyte Scale Data Warehouse AWS Global Infrastructure

10 Database RDS Dynamo DB Redshift Elasticache Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure Relational Database Service Database-as-a-Service No need to install or manage database instances Scalable and fault tolerant configurations

11 Database RDS Dynamo DB Redshift Elasticache Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure DynamoDB Provisioned throughput NoSQL database Fast, predictable, configurable performance Fully distributed, fault tolerant HA architecture Integration with EMR & Hive

12 Dynamo Consistency Writes Writes are acknowledged (committed) once they exist in at least two physical data centers Writes are persisted to SSD Reads Tunable for Application Requirements No reduction in durability or consistency in order to achieve throughput Eventually Consistent Read Stale Values reads possible Highest Throughput Strongly Consistent Read No Stale Values read Lower Potential Throughput

13 Database RDS Dynamo DB Redshift Elasticache Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure Redshift Managed Massively Parallel Petabyte Scale Data Warehouse Streaming Backup/Restore to S3 Load data from S3, DynamoDB and EMR Extensive Security Features Scale from 160 GB -> 1.6 PB Online

14 Redshift Parallelizes Everything Common BI Tools Query JDBC/ ODBC Load Leader Node Backup 1 0 GigE Mesh Restore Compute Node Resize Compute Node Compute Node

15 Exploratory Analytics Data Cleansing Advanced Data Science

16 Managed Big Data Elastic MapReduce Deployment & Administration App Services Compute Storage Networking Analytics Databas e AWS Global Infrastructure Elastic MapReduce Managed, elastic Hadoop (1.x & 2.x) cluster Integrates with S3, DynamoDB and Redshift Install End User Tools Automatically (Spark, Impala) Support for EC2 Spot Instances

17

18 Vibrant Ecosystem Pig HDFS EMR

19 Weather Insurance for Farms Challenge: Volatile weather is deadly to crops like grapes Solution: Built a predictive model based on freely available data: 150B Soil Observations 60 years of crop data 200 TB of S3 Data 1M government Doppler radar points 850K Precision Rainfall Grids Tracked 3M Daily Weather Measurements 50 EMR clusters process new data as it comes into S3 each day, continuously updating the model

20 Choose your instance types General m3 family CPU c3 family cc2.8xlarge d2 family Memory m2 family r3 family Disk/IO d2 family i2 family ETL Machine Learning Spark HDFS Try different configurations to find the optimal cost/performance balance

21 Custom Intel Xeon processors for AWS C4 = highest performing EC2 instances New EC2 Instances C4

22 The Financial Industry Regulatory Authority 30 Billion Market Events / Day Objective to react to changing Market Dynamics Amazon Elastic MapReduce & Amazon S3 $10-20M Savings by moving Platform to AWS

23 Event Processing Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure AWS Lambda Fully Managed Event Processor Node.js, Integrated AWS SDK & ImageMagick Natively Compile & Install Node.js modules Specify Runtime RAM & Timeout Automatically Scaled to support Event Volume Events from S3, Dynamo DB, Kinesis & Lambda Integrated CloudWatch Logging

24 Introducing Amazon Machine Learning SDE expertise Easily create machine learning models Visualize and optimize models Put models into production in seconds Machine Learning expertise Battle-hardened technology

25 Easy to Use, High Performance Train and optimize models on GBs of data Batch process predictions Real-time prediction API in one-click No servers to provision or manage

26 Developing with Amazon Machine Learning Build model Validate & optimize Make predictions

27 Building a Predictive Model with Amazon Machine Learning Use existing data in S3, Redshift and RDS Automatic data visualization & exploration Descriptive and summary statistics Your data doesn t have to be perfect Missing data, malformed data records, type validation

28 Model Validation and Optimization Tools

29 Making Predictions with Amazon Machine Learning Batch predictions Asynchronous predictions with trained model Real time predictions Synchronous, low latency, high throughput Mount API end-point with a single click

30 Traditional Business Intelligence OLAP Data Sources for ML

31 Managed Data Warehouse RDS Dynamo DB Redshift ElastiCache Deployment & Administration App Services Compute Storage Networking Analytics Databas e AWS Global Infrastructure Redshift Managed Massively Parallel Petabyte Scale Data Warehouse Streaming Backup/Restore to S3 Load data from S3, DynamoDB and EMR Extensive Security Features Scale from 160 GB -> 1.6 PB Online

32 Redshift lets you start small and grow big Extra Large Node (dw1.xl & dw2.xl) 3 spindles, 15GiB RAM 2 virtual cores, 10GigE Single Node (160GB SSD or 2TB Magnetic) Cluster 2-32 Nodes (320GB SSD 64TB Magnetic) 8 Extra Large Node (dw1.8xl & dw2.8xl) 24 spindles, 120GiB RAM, 1.2TB SSD or 16TB Magnetic, 16 virtual cores, 10GigE Cluster Nodes (2.4TB SSD 1.6PB Magnetic)

33 End User Reporting EMR Redshift S3 Dynamo DB

34 Ignite Your Ambition 26 Markets 3 Clearing Houses 5 Central Securities Depositories Leading Index Provider With 41,000+ Indexes Across Asset Classes And Geographies Over 10,000 Corporate Clients in 60 countries Lists more than 3,500 companies in 35 countries, representing more than $8.8 trillion in total market value 100+ DATA PRODUCT OFFERINGS supporting 2.5+ million investment professionals and users IN 98 COUNTRIES Our technology powers over 70 MARKETPLACES, regulators, CSDs and clearinghouses in over 50 COUNTRIES 34

35 NDW 1.0 Requirements Original scope was to replace on-premises warehouse with Redshift, keeping equivalent schemas and data 4-8 Billion Rows/Day Legacy limited to 1 Year Retention Must be lower cost than legacy system Legacy $1.16M/Year Must satisfy multiple security and regulatory requirements Must perform similarly to legacy warehouse under concurrent query load

36 Migration Completed On Schedule Migrated off legacy warehouse to Redshift (start to finish) in 7 man-months Redshift costs were 43% of legacy budget for the same data set (~1100 tables) Tuned queries now running faster than on legacy system Data Ingest 5.5B rows/day average for 2014 High water mark: 14B rows in 1 day Best write rates ~2.76M rows/second 450 GB/day (after compression) into Redshift 1,895 GB/day average uncompressed Currently resize clusters once a quarter (if necessary) NDW_Prod is currently growing +3 dw1.8xl nodes per quarter

37 Integrated Analytics