Gabriel Villa Architecting an Analytics Solution on AWS
Cloud and Data Architect Skilled leader, solution architect, and technical expert focusing primarily on Microsoft technologies and AWS. Passionate about open-source technologies for integration, automation, and development. Tech Community Speaker at local, regional and national conferences, and user groups. Board Member of Colorado Springs SQL Server User Group Gabriel Villa Architect, Revgen Partners /gabevilla @extofer Past Leader of PASS Application Development Virtual Chapter Founding Member and Past Organizer of Colorado Give Camp Colorado Transplant Originally from El Paso, Texas, living in Colorado Springs with his wife and kids, volunteers in the tech community and enjoys the Colorado outdoors.
Outline Cloud Database Architecture Managed Services Services Case Study Solution Architecture Final Thoughts
N-Tier Architecture Client Middle Tier Database
Cloud Database Architecture Client Middle Tier Athena S3 EMR RDS CloudSearch Database Tier ES Kinesis Redshift DynamoDB AWS Glue
What is a managed Service Hardware Procurement Installations Ongoing Maintenance Patching
Benefits of AWS Database and Analytic Services Managed Services Easy to Scale Pay only what you use Use AWS Ecosystem Analytics Development Messaging Migration Compute Database Networking Management Tools Security Artificial Intelligence
Computing Node.js Python EC2 AWS Lambda.NET Java
Relational Database Services RDS
Demo
Dynamo DB Managed NO-SQL Database NoSQL Categories (uses) KV stores Document Stores (JSON) Graph Databases (node or edges) DynamoDB
Data Warehousing Redshift Managed Petabyte-Scale Data Warehouse Massively Parallel Processing Columnar Storage Fully Elastic, grow and shrink No License, pay for what you use Persistent or Transient
Storage S3 Glacier Durable, highly scalable cloud storage 99.999999999% Durability 99.99% Availability Bucket Archive
Demo
Elasticsearch Service Distributed Text Indexing Fully Managed Log Searching ES
Streaming Data Real-Time Data Streaming And Analytics Big Data Analytics Minimal Transformation Kinesis
Data Workflow Orchestration and ETL AWS Data Pipeline AWS Glue
Data Migration One-Time Data Migration Continuous Data Replication Convert Stored Procedures Data Migration Service
Putting it all together On-Prem Application Logs ES AWS Lambda Kinesis AWS Lambda S3 AWS ML Athena AWS Glue S3 AWS Data Pipeline AWS Quicksight AWS Lambda AWS DMS Redshift
Case Study: DynamoDB at a Railroad Service Built and deployed an operational database and data mart for near-real-time reporting of sales Developed and released the solution in 6 months Used AWS technologies DynamoDB, Kinesis, Lambda and S3 Benefits Improved accuracy and single source of truth for sales data Allows decommissioning of four legacy systems Low maintenance and operation cost no servers to mange
Solution architecture Legacy Systems Booking AWS Lambda Kinesis AWS Lambda API Tickets DynamoDB Redshift RDBMS Sources AWS Data Pipeline ETL S3: Flat File AWS Lambda Kinesis AWS Lambda S3: Reporting
Machine Learning Models for Predictive Applications Follow visual tools and wizards to create models Process new data in batch, or Realtime with APIs Machine Learning
Demo
Business Analytics QuickSight Supports assorted visualizations AWS data sources or uploads Integrate with Saas Applications
Final Thoughts You don t need your entire infrastructure on the cloud to build your analytics solutions You can connect with AWS securely and with little latency Your data is secure and high availability options do exist on-prem cloud
Final Thought: Coexistence Architecture S3 corporate data center ES Lambda Kinesis Lambda ML Athena Glue Data Pipeline S3 Quicksight On-Prem Lambda Glue Redshift
Wrap Up Databases RedShift, RDS, DynamoDB Processing - Lamdba Stream Processing Kinesis Data Flow DMS, Glue and Data Pipeline Machine Learning Business Analytics QuickSight
Any Questions? /gabevilla @extofer
Thank you! /gabevilla @extofer