Experiences with Serverless Big Data

Transcription

1 Experiences with Serverless Big Data AWS Meetup Munich 2016 Markus Schmidberger, Head of Data Service Munich,

2 Key Components of our Data Service Real-Time Monitoring Enable our development teams to serve our content to our users in the best quality possible. Analytics Provide our teams access to the data to enable data-driven development of new features and products. Content Discovery Find the most relevant content for our customers and their users. 2

3 Micro-Service Architecture Data Platform - MicroService Layout Content API Content Import Service Content Discovery Service other modules CDN files CDN Log Import Service Data Management Service Metadata Service KPI & Analytics Service Data API Portal data stream data stream AdProxy Log Import Service VAS Log Import Service Data Lake Data Layer Technical Monitoring Service Real-Time Dashboards data stream Player Feedback Import Service Data Quality Service Dev / Ops Analytics Service Data Platform Access Team External Data Import Service Data Science Analytics Service Data Science UI Data Platform Monitoring Service INGEST STORE PROCESS & ANALYSE VISUALIZE & SERVE 3

4 Lambda Architecture Batch Processing KPIs for MES MES Billing Data Science CDN Bills Data Insights Real-time Real-time player monitoring Internal dashboard 4

5 Lambda Architecture AWS Lambda Graphic provided by 5

6 AWS Lambda Notification Amazon S3 Amazon S3 AWS Lambda processes the object New object uploaded Amazon DynamoDB 6

7 AWS Lambda Execution 7

8 Be serverless for ETL 8

9 Be serverless and serve data Amazon S3 AWS Lambda AWS Lambda Amazon API Gateway 9

10 Be serverless for Recommendations Recommendation Pipeline Download Page content Publisher s URL Search Recommender System Content Playlist 10

11 Be serverless everywhere Read data from Kinesis Firehose / S3 Server downtime / scheduler Load to ElasticSearch Clean ElasticSearch and Redshift Advanced Redshift monitoring EBS Snapshots 11

12 Agile DevOps Continous Delivery of Micro-services Automate all the things 12

13 Agile Cloud Deployment Glomex Cloud Deployment Tools Kumo AWS CloudFormation Ramuda AWS Lambda Yugen AWS API Gateway Tenkai AWS CodeDeploy In cooperation with Operations Team Used by other teams Simplify cloud deployments drastically Slack- and monitoring integration 13

14 Agile DevOps Cross-functional team responsible to push components to production themselves 14

15 AWS Lambda Limits 5 min 512 MB AWS Lambda Timeout AWS Lambda temp disk How to process 800MB gziped logfile? How to split compressed gzip files? Splitter using Amazon SQS and Amazon EC2 Spot Instances 15

16 More AWS Lambda limits Lambda function deployment package size (.zip/.jar file) 50 MB Total size of all the deployment packages that can be uploaded per region 75 GB CreateLogGroup 500 log groups / account / region Lambda functions have to be wired Be aware of retries 3 not configurable Traceback and error output available via CloudWatch Logs No local development environment 16

17 More Facts 20 GB 5 Billion ~100 ms < 1 min 25 GB 300 Million Per day click-stream data IN (player, vas, adproxy) Click-stream records processed per day Data freshness to S3 Data freshness to API Per day as zipped CDN log-files CDN record processed per day 17

18 More Facts 2.3 min 600 rec/sec 6 1 $ / hour Average run-time of AWS Lambda Processing time Parallel AWS Lambda functions Cost for 25 GB/day CDN processing AWS Lambda duration Redshift CPU ~ 89% % Accuracy 18

19 Key Takeaways Lambda Architecture Enrich your traditional, batch-driven BI-workflow with real-time analytics Use Lambda-Architecture as a guiding principle and adapt it to your needs 19

20 Key Takeaways 1 Analyze 3 Automate 2 Take Actions 20

21 Key Takeaways Focus on features development and robust pipelines not on infrastructure management AWS managed services provide an robust way to run complex big data infrastructures Follow best-practices provided by AWS and the community 21

22 We are hiring Data Engineers Product Owner Big Data Product Line Markus Schmidberger, Head of Data Service Munich,