Devices Device Connectivity Storage Analytics Presentation & Action. Table/Blob Storage. External Data Sources

Size: px

Start display at page:

Download "Devices Device Connectivity Storage Analytics Presentation & Action. Table/Blob Storage. External Data Sources"

Griffin Dalton
6 years ago
Views:

Stream Analytics Power BI External Data Sources { } DocumentDB HDInsight

2 Devices Device Connectivity Storage Analytics Presentation & Action Event Hubs SQL Database Machine Learning App Service Service Bus Table/Blob Storage Stream Analytics Power BI External Data Sources { } DocumentDB HDInsight Notification Hubs External Data Sources Data Factory Mobile Services BizTalk Services

3 Understand the pulse of the Organization What happened? What is happening? Why did it happen? What will happen? Past Present Future

5 Everything around us produce data Traditional Business Intelligence first collects data and analyzes it afterwards But we live in a fast paced world Offline data is unuseful

6 We work with streaming data We want to monitor and analyze data in near real time So we don t have the time to stop, copy data and analyze, but we have to work with streams of data

Sort of queries on data warehouse Data in motion

7 Batch Analytics Data Parking into Relational database Bigdata notion velocity verity volume Sort of queries on data warehouse Data in motion & live event Cost effective way Business competitive age

9 Intake millions of events per second At variable loads Transform, augment, correlate, temporal operations Elasticity of the cloud for scale out No hardware (PaaS offering) Rapid development

11 Monitoring and Troubleshooting of solution Develop solutions and infrastructure for increasing scale with business growth Develop solutions to manage resiliency, such as infrastructure failures Develop solutions to integrate with other components like ML, BI etc Develop solution (code) for ingress, processing and egress Infrastructure Procure and setup Development and operations resources Time

12 Monitoring and Troubleshooting of solution Develop solutions and infrastructure for increasing scale with business growth Develop solutions to manage resiliency, such as infrastructure failures Develop solutions to integrate with other components like ML, BI etc Develop solution (code) for ingress, processing and egress From Event or Data Streams to Real Time Insights in less time with less people resources Infrastructure Procure and setup

13 End-to-End Architecture Overview Azure Stream Analytics Temporal Semantics Guaranteed delivery Guaranteed up time Event Inputs - Event Hub - Azure Blob Reference Data - Azure Blob Transform - Temporal joins - Filter - Aggregates - Projections - Windows - Etc. Enrich Correlate Outputs - SQL Azure - Azure Blobs - Event Hub Azure Storage Data Source Collect Process Deliver Consume

14 Aerocrine has 26 sales territories staffed, 4 regional managers

15 Rapid Development Decrease bar to create Stream Processing Solutions via SQL-like Language Easily filter, project, aggregate, join streams, add static data with streaming data, detect patterns or lack of patterns with a few lines of SQL Built-in temporal semantics Development and debugging experience through Azure Portal Manage out-of-order events & actions on late arriving events via configurations

17 SELECT count(*), Topic FROM Tweets GROUP BY Topic, TumblingWindow(second, 5)

Pain Points with other Streaming Solutions Not an end to end solution Hard to develop Need expertise and special skills Costs lot of money on Development @ApplicationAnnotation(name="WordCountDemo")

$txt"; private Locality locality = null; } @Override public void populatedag(dag dag, Configuration conf) { locality = Locality.CONTAINER_LOCAL; WordCountInputOperator input = dag.$

18 Pain Points with other Streaming Solutions Not an end to end solution Hard to develop Need expertise and special skills Costs lot of money on public class Application implements StreamingApplication { protected String filename = "com/datatorrent/demos/wordcount/samplefile.txt"; private Locality locality = null; public void populatedag(dag dag, Configuration conf) { locality = Locality.CONTAINER_LOCAL; WordCountInputOperator input = dag.addoperator("wordinput", new WordCountInputOperator()); input.setfilename(filename); UniqueCounter<String> wordcount = dag.addoperator("count", new UniqueCounter<String>()); dag.addstream("wordinput-count", input.outputport, wordcount.data).setlocality(locality); ConsoleOutputOperator consoleoperator = dag.addoperator("console", new ConsoleOutputOperator()); dag.addstream("count-console",wordcount.count, consoleoperator.input); }

No code compilation, easy to author and deploy Brings together event streams, reference data and machine learning extensions All operators respect, and some use, the temporal

20 No code compilation, easy to author and deploy Brings together event streams, reference data and machine learning extensions All operators respect, and some use, the temporal properties of events These should (mostly) look familiar if you know relational databases Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation

Our toll station has multiple toll booths, where a sensor placed on top of the booth scans an RFID card affixed to the windshield of the vehicles as they pass the toll booth.

21 Our toll station has multiple toll booths, where a sensor placed on top of the booth scans an RFID card affixed to the windshield of the vehicles as they pass the toll booth. The passage of vehicles through these toll stations can be modelled as event streams over which interesting operations can be performed. Toll Id 1 2 EntryTime LicensePlate State Make Model :01: :02: Vehicle Type Vehicle Weight JNB 7001 NY Honda CRV YXZ 1001 NY Toyota Camry Toll Tag Toll Id ExitTime LicensePlate T12:03: Z JNB T12:03: Z YXZ 1001

22 Projections Show me the Toll Id and Vehicle Weight in Tons for all vehicles passing through the Toll Booth 1, 1450, VW, Golf, ( ) 2, 1230, Toyota, Camry, ( ) 1, 2400, VW, Passat, ( ) 1, 980, Ford, Fiesta, ( ) SELECT TollId, VehicleWeight / 1000 AS Tons FROM EntryStream 1, , , , 0.980

23 Filters Show me the Model of vehicles manufactured by Volkswagen 1, 1450, VW, Golf, ( ) 2, 1230, Toyota, Camry, ( ) 1, 2400, VW, Passat, ( ) 1, 980, Ford, Fiesta, ( ) SELECT Model FROM EntryStream WHERE Make = "VW" Golf Passat

24 Tumbling Windows How many vehicles entered each toll both every 5 minutes? SELECT TollId, COUNT(*) FROM EntryStream GROUP BY TollId, TumblingWindow(minute,5)

25 Aggregate functions Scalar functions Date and time: String: Types Type Description bigint Integers in the range -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807). float nvarchar(max) datetime Floating point numbers in the range E+308 to -2.23E-308, 0, and 2.23E-308 to 1.79E+308. Text values, comprised of Unicode characters. Note: A value other than max is not supported. Defines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock and relative to UTC (time zone offset 0).

Welcome to SQL Saturday Denmark

Welcome to SQL Saturday Denmark Complex Event Processing with Azure Stream Analytics Azure Stream Analytics is a fully managed service complex event processing in the cloud It's a big data analytics service