Streaming Data: The Opportunity & How to Work With It

Transcription

1 Streaming Data: The Opportunity & How to Work With It Roger Barga, GM Amazon Kinesis April , Amazon Web Services, Inc. or its Affiliates. All rights reserved.

2 Interest in and demand for stream data processing is rapidly increasing * * Understatement of the year

3 Why? Most data is produced continuously user-identifier frank [10/Oct/2000:13:55: ] "GET /apache_pb.gif HTTP/1.0" Common Log Entry SeattlePublicWater/Kinesis/123/Realtime MQTT Record { "payerid": "Joe", "productcode": "AmazonS3", "clientproductcode": "AmazonS3", "usagetype": "Bandwidth", "operation": "PUT", "value": "22490", "timestamp": " " } Metering Record <R,AMZN,T,G,R1> NASDAQ OMX Record <165> T22:14:15.003Z mymachine.example.com evntslog - ID47 [examplesdid@32473 iut="3" eventsource="application" eventid="1011"][examplepriority@32473 class="high"] Syslog Entry

4 Why? Time is money Perishable Insights (Forrester) Hourly server logs: how your systems were misbehaving an hour ago Weekly / Monthly Bill: What you spent this past billing cycle? Daily fraud reports: tells you if there was fraud yesterday CloudWatch metrics: what just went wrong now Real-time spending alerts/caps: guaranteeing you can t overspend Real-time detection: blocks fraudulent use now

5 Why? Time is money Perishable Insights (Forrester) A more efficient implementation Most Big Data deployments process continuously generated data (batched)

6 Amazon Kinesis: Streaming Data Done the AWS Way Makes it easy to capture, deliver, and process real-time data streams Easy to provision, deploy, and manage Elastically scalable Real-time latencies Pay as you go, no up-front costs Right services for your specific use cases

7 Amazon Kinesis: Streaming data made easy Services make it easy to capture, deliver and process streams on AWS Kinesis Streams For Technical Developers Kinesis Firehose For all developers, data scientists Kinesis Analytics For all developers, data enginers, data scientists Build your own custom applications that process or analyze streaming data Easily load massive volumes of streaming data into S3 and Redshift Easily analyze data streams using standard SQL queries

8 Amazon Kinesis Streams Build your own data streaming applications Easy administration: Simply create a new stream, and set the desired level of capacity with shards. Scale to match your data throughput rate and volume. Build real-time applications: Perform continual processing on streaming big data using Kinesis Client Library (KCL), Apache Spark/Storm, AWS Lambda, and more. Low cost: Cost-efficient for workloads of any scale.

9 Kinesis Streaming Data Ingestion Streams are made of Shards Each Shard ingests data up to 1MB/sec, and up to 1000 TPS Producers use PUT call to store data in a Stream: PutRecord {Data, PartitionKey, StreamName} Each shard emits 2MB/sec All data is stored for 24 hours Scale Kinesis streams by adding or removing Shards Replay data from stream (log)

10 Kinesis Producer Library Highly configurable library to write to Kinesis Collects records and uses PutRecords for high throughput writes Integrates seamlessly with the Amazon Kinesis Client Library (KCL) to de-aggregate batched records Submits Amazon CloudWatch metrics on your behalf to provide visibility into producer performance

11 Extended Retention in Kinesis Default 24 Hours but configurable to 7 days 2 New APIs IncreaseStreamRetentionPeriod(String StreamName, int RetentionPeriodHours) DecreaseStreamRetentionPeriod(String StreamName, int RetentionPeriodHours) Use it in one of two-modes As always-on extended retention Raise retention period in response to operational event or planned maintenance of record processing application to extend retention.

12 25 40ms ms Aggregate and archive to S3 Millions of sources producing 100s of terabytes per hour Front End Authentication Authorization AZ AZ AZ Durable, highly consistent storage replicates data across three data centers (availability zones) Amazon Web Services Inexpensive: Inexpensive: $0.014 $0.028 per 1,000,000 per million PUT putspayload Units Ordered stream of events supports multiple readers Real-time dashboards and alarms Machine learning algorithms or sliding window analytics Custom-built Streaming Applications (KCL) Real-Time Streaming Data Ingestion Aggregate analysis in Hadoop or a data warehouse

13 Building Applications: Kinesis Client Library For fault-tolerant, scalable stream processing applications Open Source Java client library, also available for Python, Ruby, Node.JS,.NET. Available on Github Connects to the stream and enumerates the shards Instantiates a record processor for every shard it manages Pulls data records from the stream Pushes the records to the corresponding record processor

14 Building Applications: Kinesis Client Library For fault-tolerant, scalable stream processing applications Open Source Java client library, also available for Python, Ruby, Node.JS,.NET. Available on Github Connects to the stream and enumerates the shards Instantiates a record processor for every shard it manages Pulls data records from the stream Pushes the records to the corresponding record processor Checkpoints processed records Balances shard-worker associations when the worker instance count changes Balances shard-worker associations when shards are split or merged

15 Real-Time Streaming Data with Kinesis Streams 1 TB+/day game data analyzed in real-time Gaming 5 billion events/wk from connected devices IoT 17 PB of game data per season Entertainment 100 billion ad impressions/day, 30 ms response time Ad Tech 100 GB/day click streams 250+ sites Enterprise 50 billion ad impressions/day sub-50 ms responses Ad Tech 17 million events/day Technology 1 billion transactions per day Bitcoin

16 Amazon Kinesis: Streaming data made easy Services make it easy to capture, deliver and process streams on AWS Kinesis Streams For Technical Developers Kinesis Firehose For all developers, data scientists Kinesis Analytics For all developers, data enginers, data scientists Build your own custom applications that process or analyze streaming data Easily load massive volumes of streaming data into S3 and Redshift Easily analyze data streams using standard SQL queries

17 Amazon Kinesis Firehose Capture and submit streaming data to Firehose Firehose loads streaming data continuously into S3 and Redshift Analyze streaming data using your favorite BI tools Zero Admin: Capture and deliver streaming data into S3, Redshift, and more without writing an application or managing infrastructure Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into S3, and other destinations in as little as 60 secs, set up in minutes Seamless elasticity: Seamlessly scales to match data throughput

18 Amazon Kinesis Firehose 3 Simple Concepts 1. Delivery Stream: The underlying entity of Firehose. Use Firehose by creating a delivery stream to a specified destination and send data to it. You do not have to create a stream or provision shards. You do not have to specify partition keys. 2. Records: The data producer sends data blobs as large as 1,000 KB to a delivery stream. That data blob is called a record. 3. Data Producers: Producers send records to a Delivery Stream. For example, a web server sending log data to a delivery stream is a data producer.

19 Amazon Kinesis Firehose Console Experience Unified Console Experience for Firehose and Streams

20 Amazon Kinesis Firehose Console Experience (S3) Create fully managed resources for delivery without building an app

21 Amazon Kinesis Firehose Console Experience Configure data delivery options simply using the console

22 Amazon Kinesis Agent Software agent makes submitting data to Firehose easy Monitors files and sends new data records to a delivery stream Handles file rotation, checkpointing, and retry upon failures Delivers all data in a reliable, timely, and simple manner Emits AWS CloudWatch metrics to help you better monitor and troubleshoot the streaming process Supported on Amazon Linux AMI with version or later, or Red Hat Enterprise Linux version 7 or later. Install on Linux-based server environments such as web servers, front ends, log servers, and more Also enabled for Kinesis Streams

23 Amazon Kinesis Firehose Pricing Simple, pay-as-you-go and no up-front costs Dimension Value Per 1 GB of data ingested $0.035

24 Hearst s wonderful world of streaming data Digital Media Publication brands Mobile Apps Re-targeting platforms

25 Hearst Streaming Data Architecture Clickstream API Ready Data Users to Hearst Properties Node.JS App- Proxy Kinesis Streams ETL on EMR Amazon Redshift Data Science Application Buzzing API LATENCY THROUGHPUT Milliseconds 100GB/day Kinesis Firehose Agg Data Models S3 Storage 30 seconds 5GB/day 100 seconds 1G/day 5 seconds 1G/day

26 AWS Endpoint Amazon Kinesis Firehose Fully Managed Service for Delivering Data Streams into AWS Destinations Data Sources No Partition Keys No Provisioning End to End Elastic [Batch, Compress, Encrypt] S3 Redshift

27 Amazon Kinesis Streams is a service for workloads that requires custom processing, per incoming record, with sub-1 second processing latency, and a choice of stream processing frameworks. Amazon Kinesis Firehose is a service for workloads that require zero administration, ability to use existing analytics tools based on S3 or Redshift, and a data latency of 60 seconds or higher.

28 Real-Time Analytics Value of insights on some data degrades quickly with time, like stock markets, traffic, device telemetry, patient monitoring, etc. Our customers asked for a way to produce results fast Static Queries, but need very fast output (alerts, real-time control) Ability to transform streaming data and deliver enhanced data to other destinations (services & storage) Dynamic and interactive queries (data exploration)

29 Amazon Kinesis: Streaming data made easy Services make it easy to capture, deliver and process streams on AWS Kinesis Streams For Technical Developers Kinesis Firehose For all developers, data scientists Kinesis Analytics For all developers, data enginers, data scientists Build your own custom applications that process or analyze streaming data Easily load massive volumes of streaming data into S3 and Redshift Easily analyze data streams using standard SQL queries

30 Amazon Kinesis Analytics Analyze data streams continuously with standard SQL Announcement Only! Connect to Kinesis streams, Firehose delivery streams Run standard SQL queries against data streams Kinesis Analytics can send processed data to analytics tools so you can create alerts and respond in real-time Apply SQL on streams: Easily connect to data streams and apply existing SQL skills. Build real-time applications: Perform continual processing on streaming big data with sub-second processing latencies Scale elastically: Elastically scales to match data throughput without any operator intervention.

31 Scenarios for Streaming Data Analytics Windowed analytics such as running totals, weighted moving averages using sliding and tumbling windows Streaming ETL to parse, validate, and enrich streaming data, and then aggregate and filter before sending to persistent stores like S3, Redshift, ElasticSearch, and more Real-time alerts on data streams using simple thresholds or advanced machine learning algorithms like anomaly detect Real-time visualizations on calculated key performance indicators in near real-time and more sophisticated use cases such as temporal analysis, detecting event sequences, complex event processing, advanced analytics.

32 Amazon Kinesis Analytics Service Overview

33 Use ANSI SQL to build real-time streaming applications Connect to Streaming Input Data Select Kinesis Streams or Firehoses as input data for your SQL code Easily write SQL code to process streaming data Author applications that perform simple aggregations to sophisticated multi-step pipelines with advanced analytical functions Continuously deliver SQL results Configure one to many outputs for your processed data for real-time alerts, visualizations, and batch analytics

34 Connect to Streaming Input Data Source Single streaming input, Kinesis Stream or Firehose Single reference input up to 1 GB from S3 Input formats include JSON, CSV, variable column (TSV, pipe-delimited); complex, nested JSON structures Each input has a schema Schema is inferred but you can edit Deep nesting (2+ levels) supported by specifying row paths of desired fields Multiple record types are supported through canonical schema and/or by manipulating structures in SQL using string functions

35 Easily write SQL to process streaming data Build streaming application with SQL statements Extensions to SQL standard to work seamlessly with streaming data (STREAM, Windows, ROWTIME) Robust SQL support including large number of functions including: Simple mathematical operators (AVG, STDEV, etc.) String manipulations (SUBSTRING, POSITION) Advanced analytics (random sampling, anomaly detection) Support for at-least-once processing semantics

36 Real-Time Analytics Patterns Simple counting (e.g. failure count) Counting with Windows ( e.g. failure count every hour) Preprocessing: filter, transform (e.g. data cleanup) Alerts, thresholds (e.g. Alarm on high temperature) Data correlation, detect missing events, detect erroneous data (e.g. detecting failed sensors) Joining event streams (e.g. detect a hit on soccer ball) Merge with data in a databaseupdate data conditionally

37 Realtime Analytics Patterns (contd.) Detect Event Sequence Patterns (e.g. small transaction followed by large transaction) Tracking - follow some related entity s state in space, time etc. (e.g. location of airline baggage, vehicle tracking, ) Detect trends Rise, turn, fall, Outliers, Complex trends like triple bottom etc., (e.g. algorithmic trading, SLA, load balancing) And more.

38 Continuously deliver SQL results Up to five outputs, including S3, Redshift through Firehose, Kinesis Streams, and more... Firehose allows Kinesis Analytics to separate processing and delivery of enriched streaming data Delivery speed will be heavily dependent upon your SQL queries (i.e. simple ETL versus 10 minute aggregations) Other destination(s) will be included at GA. Output formats include JSON, CSV, variable column (TSV, pipe-delimited)

39 Questions

40