Gabriel Villa. Architecting an Analytics Solution on AWS

Transcription

1 Gabriel Villa Architecting an Analytics Solution on AWS

2 Cloud and Data Architect Skilled leader, solution architect, and technical expert focusing primarily on Microsoft technologies and AWS. Passionate about open-source technologies for integration, automation, and development. Tech Community Speaker at local, regional and national conferences, and user groups. Board Member of Colorado Springs SQL Server User Group Gabriel Villa Architect, Revgen Partners Past Leader of PASS Application Development Virtual Chapter Founding Member and Past Organizer of Colorado Give Camp Colorado Transplant Originally from El Paso, Texas, living in Colorado Springs with his wife and kids, volunteers in the tech community and enjoys the Colorado outdoors.

3 Outline Cloud Database Architecture Managed Services Services Case Study Solution Architecture Final Thoughts

4 N-Tier Architecture Client Middle Tier Database

5 Cloud Database Architecture Client Middle Tier Athena S3 EMR RDS CloudSearch Database Tier ES Kinesis Redshift DynamoDB AWS Glue

6 What is a managed Service Hardware Procurement Installations Ongoing Maintenance Patching

7 Benefits of AWS Database and Analytic Services Managed Services Easy to Scale Pay only what you use Use AWS Ecosystem Analytics Development Messaging Migration Compute Database Networking Management Tools Security Artificial Intelligence

8 Computing Node.js Python EC2 AWS Lambda.NET Java

9 Relational Database Services RDS

10 Demo

11 Dynamo DB Managed NO-SQL Database NoSQL Categories (uses) KV stores Document Stores (JSON) Graph Databases (node or edges) DynamoDB

12 Data Warehousing Redshift Managed Petabyte-Scale Data Warehouse Massively Parallel Processing Columnar Storage Fully Elastic, grow and shrink No License, pay for what you use Persistent or Transient

13 Storage S3 Glacier Durable, highly scalable cloud storage % Durability 99.99% Availability Bucket Archive

14 Demo

15 Elasticsearch Service Distributed Text Indexing Fully Managed Log Searching ES

16 Streaming Data Real-Time Data Streaming And Analytics Big Data Analytics Minimal Transformation Kinesis

17 Data Workflow Orchestration and ETL AWS Data Pipeline AWS Glue

18 Data Migration One-Time Data Migration Continuous Data Replication Convert Stored Procedures Data Migration Service

19 Putting it all together On-Prem Application Logs ES AWS Lambda Kinesis AWS Lambda S3 AWS ML Athena AWS Glue S3 AWS Data Pipeline AWS Quicksight AWS Lambda AWS DMS Redshift

20 Case Study: DynamoDB at a Railroad Service Built and deployed an operational database and data mart for near-real-time reporting of sales Developed and released the solution in 6 months Used AWS technologies DynamoDB, Kinesis, Lambda and S3 Benefits Improved accuracy and single source of truth for sales data Allows decommissioning of four legacy systems Low maintenance and operation cost no servers to mange

21 Solution architecture Legacy Systems Booking AWS Lambda Kinesis AWS Lambda API Tickets DynamoDB Redshift RDBMS Sources AWS Data Pipeline ETL S3: Flat File AWS Lambda Kinesis AWS Lambda S3: Reporting

22 Machine Learning Models for Predictive Applications Follow visual tools and wizards to create models Process new data in batch, or Realtime with APIs Machine Learning

23 Demo

24 Business Analytics QuickSight Supports assorted visualizations AWS data sources or uploads Integrate with Saas Applications

25 Final Thoughts You don t need your entire infrastructure on the cloud to build your analytics solutions You can connect with AWS securely and with little latency Your data is secure and high availability options do exist on-prem cloud

26 Final Thought: Coexistence Architecture S3 corporate data center ES Lambda Kinesis Lambda ML Athena Glue Data Pipeline S3 Quicksight On-Prem Lambda Glue Redshift

27 Wrap Up Databases RedShift, RDS, DynamoDB Processing - Lamdba Stream Processing Kinesis Data Flow DMS, Glue and Data Pipeline Machine Learning Business Analytics QuickSight

28 Any Questions?

29 Thank you!