Welcome to SQL Saturday Denmark

Transcription

1 Welcome to SQL Saturday Denmark

2 Complex Event Processing with Azure Stream Analytics Azure Stream Analytics is a fully managed service complex event processing in the cloud It's a big data analytics service for the Internet of Things (IoT)

3 Thanks you our PLATINUM sponsors

4 Thanks you our GOLD and SILVER sponsors

5 About the speaker Mihail Mateev is a Technical Consultant, Community enthusiast, PASS RM for CEE and chapter lead, Microsoft Azure MVP Solutions Architect, Technical PM and Senior Technical Evangelist at Strypes (Technical storyteller ) Mihail works in various areas related to Microsoft technologies : Windows Platform, ASP.Net MVC, MS SQL Server and Microsoft Azure 5

6 Agenda: Introduction to Azure Stream Analytics Key capabilities Business advantages for choosing Azure Stream Analytics Scenarios & use cases Telemetry & log analysis via dashboards Event archival for future processing 6

7 What are customers wanting to do? 7

8 Introduction What is Azure Stream Analytics? Azure Stream Analytics is SaaS ( software as a service ) 8

9 Introduction What is Azure Stream Analytics? But, Azure Stream Analytics is fast ( low latency ) 9

10 Introduction What is actually Azure Stream Analytics? А fully managed service providing low-latency, highly available, scalable, complex event processing over streaming data in the cloud A big data analytics service for the Internet of Things (IoT) that enables developers to combine streams of data with historic records or reference data 10

11 Introduction What is best application of Azure Stream Analytics? Implementation of real time channels Bet fit for emergency solutions / IoT sysmes 11

12 The IoT Value Chain 12

13 The IoT Data Streams 13

14 14

15 Stream Analytics How do customers create a real-time streaming solution? 15

16 Key capabilities Ease of use Supports a simple, declarative query model for describing transformations SQL variant is selected as the domain-specific language (DSL) Insulates customers from the significant technical complexities underlying our streaming system 16

17 Key capabilities Scalability Capable of handling high event throughput of up to 1GB/second. Integration with Azure Event Hubs, IoT huns Allows the solution to ingest millions of events per second coming from connected devices, clickstreams, and log files 17

18 Key capabilities Reliability, repeatability and quick recovery A managed service in the cloud Prevents data loss and provides business continuity in the presence of node failures through its built-in recovery and check pointing capabilities Persists state to optimize resumption from node failure 18

19 Key capabilities Low latency The communication model is a pull-based, adaptive batching model that operates based on configured timeouts and size limits Even with high throughput, the system is able to provide low latencies 19

20 Key capabilities Reference data Stream Analytics provides users the ability to specify and use reference data Reference data should be Historical data or data that changes less frequently over time 20

21 Real-time analytics stack 21

22 Azure Stream Analytics More About Azure Stream Analytics. 22

23 Azure Stream Analytics Guaranteed events delivery Guaranteed not to lose events or incorrect output Preserves event order on per-device basis Guaranteed business continuity Guaranteed uptime (three nines of availability) Auto-recovery from failures Built in state management for fast recovery 23

24 Azure Stream Analytics Elasticity of the cloud for scale up or scale down Spin up any number of resources on demand Scale from small to large when required Distributed, scale-out architecture Low startup costs Provision and run Streaming solution for as low as $25/month Ability to incrementally add resources Reduce costs when business needs changes 24

25 Azure Stream Analytics Stream Processing Solutions via SQL-like Language Filter, project, aggregate, join streams, add static data with streaming data Development and debugging experience via Azure Portal Manage out-of-order events & actions on late arriving events via configurations 25

26 Azure Stream Analytics Built-in monitoring View your system s performance at a glance Help you find the cost-optimal way of deployment 26

27 End-to-End Architecture Overview 27

28 Telemetry & log analysis via dashboards 28

29 Event archival for future processing /blob storage / 29

30 Stream Analytics Input Type DocumentDB is not supported! 30

31 Azure Stream Analytics Pricing Stream analytics is not expensive (but it depends) Currently, Microsoft prices Stream Analytics by the volume of processed data and the number of stream units used to process the data, at a per-hour rate. 31

32 Azure Stream Analytics Pricing A stream unit is a compute capacity (CPU, memory, throughput), with a maximum throughput of 1 MB/s. Stream Analytics imposes a default quota of 12 streaming units per region, but requires no start-up or termination fees 32

33 Azure Stream Analytics Pricing Customers pay only for what they use, based on the following pricing structure: USD Data volume: $0.001/GB Streaming unit: $0.031/hour EUR Data volume: /GB Streaming unit: \0.0261/hour 33

34 What is Complex Event Processing? 34

35 What is Complex Event Processing? Description 1: Event processing is a method of tracking and analyzing (processing) streams of information (data) about things that happen (events) deriving a conclusion from them. 35

36 What is Complex Event Processing? CEP, is event processing that combines: data from multiple sources to infer events or patterns that suggest more complicated circumstances. The goal of complex event processing : to identify meaningful events (such as opportunities or threats) to respond to them as quickly as possible :? 36

37 What is Complex Event Processing? CEP can provide an organization with the capability to define, manage and predict in complex, heterogeneous networks: Events Situations exceptional conditions opportunities and threats 37

38 Stream Analytics and Complex Event Processing Complex Event Processing is presented via Stream Analytics job A Stream Analytics job includes all of the following: 1. One or more input sources 2. A query over an incoming data stream 3. An output target 38

39 Complex Event Processing Stream Analytics Job main elements: Input Output Queries Stream Analytics Rest API 39

40 Complex Event Processing Inputs Data stream: Each Stream Analytics job definition must contain at least one data stream input source to be consumed and transformed by the job Event Hubs IoT Hubs Azure Blob storage You can have many data stream input sources for every Stream Analytics job!!! 40

41 Complex Event Processing Inputs Reference data: Stream Analytics also supports a second type of input source: reference data: auxiliary data used for performing correlation and lookups the data here is usually static or infrequently changing Azure Blob storage blobs are limited to 50MB in size For blob you should use the {date} and {time} tokens inside the path pattern You can have only 1 reference data sources for every Stream Analytics job!!! 41

42 Complex Event Processing Inputs How to use reference data: 42

43 Complex Event Processing Inputs Serialization: Currently supported serialization formats are JSON, CSV, and Avro ( Apache Avro / Apache s Hadoop Project ) for data streams and CSV or JSON for reference data. Generated properties Depending on the input type used in the job, some additional fields with event metadata will be generated. These fields can be queried against just like other input columns 43

44 Complex Event Processing Inputs Generated properties 44

45 Azure Stream Analytics Input Options IoT hubs Event hubs Provides the ability to do realtime analysis. Eventhubs DO use blobs as storage internally AMQP 1.0 connections Azure Blobs The records in the blob should contain timestamps BLOB input - you specify which blob container to use and by default the SA engine every blob will be read only once 45

46 Azure Stream Analytics Input Options 46

47 Complex Event Processing Outputs Output Includes also Power BI: 47

48 Streaming Units Streaming Units (SUs) represent the resources and power to execute a job. SUs provide a way to describe the relative event processing capacity based on a blended measure of CPU, memory, and read and write rates Each streaming unit corresponds to roughly 1MB/second of throughput. 48

49 Configuring Streaming Units Each Azure Stream Analytics job needs a minimum of one streaming unit, which is the default for all jobs The number of streaming units that a job can utilize depends on the partition configuration for the inputs and the query defined for the job. 49

50 Configuring Streaming Units 50

51 Configuring Streaming Units Calculate the max streaming units for a job The number of streaming units that a job can utilize depends on the partition configuration for the inputs and the query defined for the job. To add additional streaming units, a step must be partitioned. Each partition can have six streaming units. 51

52 Configuring Streaming Units Calculate the max streaming units for a job 52

53 Azure Stream Analytics. Azure Stream Analytics Samples, Scenarios & Use Cases 53

54 Sample Queries Let s count tweets by topic SELECT count(*), Topic FROM Tweets GROUP BY Topic, TumblingWindow(second, 5) 54

55 Azure Stream Analytics. Azure Stream Analytics Query Language 55

56 Azure Stream Analytics. You write declarative queries in SQL Unified programming model Brings together event streams, reference data and machine learning extensions Temporal Semantics All operators respect, and some use, the temporal properties of events Built-in operators and functions These should (mostly) look familiar if you know relational databases Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation 56

57 Azure Stream Analytics Query Language Our toll station has multiple toll booths, where a sensor placed on top of the booth scans an RFID card affixed to the windshield of the vehicles as they pass the toll booth. The passage of vehicles through these toll stations can be modelled as event streams over which interesting operations can be performed. Toll Id 1 2 EntryTime LicensePlate Stat Vehicle Vehicle Make Model e Type Weight Toll Tag :01: JNB 7001 NY Honda CRV :02: YXZ 1001 NY Toyota Camry Toll Id ExitTime LicensePlate T12:03: Z JNB T12:03: Z YXZ /19/

58 Azure Stream Analytics Query Language Projections Show me the Toll Id and Vehicle Weight in Tons for all vehicles passing through the Toll Booth 1, 1450, VW, Golf, ( ) 2, 1230, Toyota, Camry, ( ) 1, 2400, VW, Passat, ( ) 1, 980, Ford, Fiesta, ( ) SELECT TollId, VehicleWeight / 1000 AS Tons FROM EntryStream 58

59 Azure Stream Analytics Query Language Filters Show me the Model of vehicles manufactured by Volkswagen 1, 1450, VW, Golf, ( ) 2, 1230, Toyota, Camry, ( ) 1, 2400, VW, Passat, ( ) 1, 980, Ford, Fiesta, ( ) SELECT Model FROM EntryStream WHERE Make = "VW" 59

60 Azure Stream Analytics Query Language Tumbling Windows How many vehicles entered each toll both every 5 minutes? SELECT TollId, COUNT(*) FROM EntryStream GROUP BY TollId, TumblingWindow(minute,5) 60

61 Azure Stream Analytics Query Language Built-in functions and supported types Aggregate functions Count, Min, Max, Avg, Sum Scalar functions Cast Date and time: Datename, Datepart, Day, Month, Year, Datediff, Dateadd String: Len, Concat, Charindex, Substring, Patindex Types 61

62 ASA Query Language Reference Events and Time From event flow SELECT * FROM SensorReadings TIMESTAMP BY time From comutations and aggregations SELECT System.Timestamp AS Time FROM SensorReadings 62

63 ASA Query Language Reference Windowing ASA supports three types of windows: 1. Tumbling 2. Hopping 3. Sliding 63

64 ASA Query Language Reference Tumbling Windows Tumbling windows are a series of fixed-sized, non-overlapping and contiguous time intervals. SELECT sensorid, COUNT(*) AS Count FROM SensorReadings TIMESTAMP BY time GROUP BY sensorid, TumblingWindow(second, 10) 64

65 ASA Query Language Reference Hopping Windows Hopping windows model scheduled overlapping windows. 3 parameters: the timeunit, the windowsize (and the hopsize SELECT sensorid, COUNT(*) AS Count, AVG(temp) FROM SensorReadings TIMESTAMP BY time GROUP BY sensorid, HoppingWindow(second, 10, 5) 65

66 ASA Query Language Reference Hopping Windows Syntax HOPPINGWINDOW ( timeunit, windowsize, hopsize, [offsetsize] ) HOPPINGWINDOW ( Duration( timeunit, windowsize ), Hop (timeunit, windowsize ), [Offset(timeunit, offsetsize)]) 66

67 ASA Query Language Practices Hopping Windows hopping windows model scheduled overlapping windows. A hopping window specification consist of three parameters: timeunit, windowsize hopsize (by how much each window moves forward relative to the previous one). Additionally,offsetsize may be used as an optional fourth parameter. 67

68 ASA Query Language Practices Sliding Windows windows of a given length and output events for cases when the content of the window actually changes: fixed length, position depends on condition SELECT sensorid, MIN(temp) as temp FROM SensorReadingsTIMESTAMP BY time GROUP BY sensorid, SlidingWindow(second, 10) HAVING MIN(temp) > 75 68

69 ASA Query Language Practices Joining Multiple Streams JOIN in ASA is used to combine records from two or more input sources: SELECT s1.time, s1.dspl, s1.hmdt as previoushmdt, s2.hmdt as newhmdt, datediff(ss, s1.time, s2.time) as secondsapart FROM SensorData s1 timestamp by time JOIN SensorData s2 timestamp by time ON s1.dspl = s2.dspl AND DATEDIFF(s, s1, s2) BETWEEN 0 AND 5 WHERE (s2.hmdt - s1.hmdt >=.1) or (s1.hmdt - s2.hmdt >=.1) 69

70 ASA Query Language Practices Reference Data JOIN Performing lookups or correlations: SELECT SensorReadings.sensorID, SensorReadings.temp FROM SensorReadings JOIN thresholdrefdata ON SensorReadings.sensorID = thresholdrefdata.sensorid WHERE SensorReadings.temp > thresholdrefdata.value 70

71 ASA Query Language Practices Multiple Outputs ASA supports multiple computations and output targets in a single job You cannot use SELECT INTO in a WITH clause. For example, INTO clause can only be used in the out-most subquery: SELECT * INTO outputlog FROM SensorReadings SELECT * INTO outputtempalert FROM SensorReadings WHERE temp > 75 71

72 ASA Query Language Practices WITH Clause Organizing a complex ASA query is to break it up into several parts using the WITH clause. WITH NormalReadings AS ( SELECT * FROM Sensor WHERE Reading < 100 AND Reading > 0 ), Averages AS ( SELECT SensorId, AVG(Reading) as AvgNormalReading FROM NormalReadings GROUP BY SensorId, TumblingWindow(minute, 1) ), BadAverages AS ( SELECT * FROM Averages WHERE AvgNormalReading < 10 ) SELECT * INTO outputalerts FROM BadAverages 72

73 Azure Stream Analytics Azure Stream Analytics Application 73

74 Azure Stream Apalytics Application IoT Solutions The major application of ASA is for IoT solutions ASA is not applicable all IoT Solutions ( like Industrial Automation) Hi speed ( very fast response) Keep longer historical data 74

75 Azure Stream Analytics Application - IoT 75

76 Azure Stream Analytics Application - IoT 76

77 Azure Stream Analytics Application - IoT Traditional approach 77

78 Azure Stream Analytics Application - IoT An approach using Azure Stream Analytics 78

79 Azure Stream Analytics Application - IoT 79

80 Azure Stream Analytics Q & A 80

81 Azure Stream Analytics Thank you! 81

82 Please review the event and sessions EVENT SESSION INSERT QR CODE FROM WORD DOCUMENT and CHANGE THE URL AS WELL /19/2016 Footer Goes Here