Event Sponsors. Expo Sponsors. Expo Light Sponsors

Transcription

1

2 Event Sponsors Expo Sponsors Expo Light Sponsors

3 IoT for the BI professional David L. Bojsen - Principal Architect

4 What to expect Level 200 session Which basically means PowerPoint and talking Enthusiastic approach to (I)IoT Want to spread the IoT-flu

5 What is IoT

6 IoT The Internet of Things (IoT) is the network of physical devices, vehicles, home appliances and other items embedded with electronics, software, sensors, actuators, and connectivity which enables these objects to connect and exchange Data Source: en.wikipedia.org/wiki/internet_of_things

7 What is IIoT

8 IIoT The term Industrial Internet of things (IIoT) is often encountered in the manufacturing industries, referring to the industrial subset of the IoT Source: en.wikipedia.org/wiki/internet_of_things

9 Internet of Things opportunity

10 Examples of IIoT scenarios Auto

11 What does that mean to us? Auto

12 But not your typical datasource IoT sensor data typically consists of time base measurements Storing, indexing, querying, analyzing, visualizing can be challenging Especially considering IoT scale Devices Storage

13 IoT Architecture

14 IoT Solutions Have a Common Pattern Things Insights Actions

15 A more realistic view Things Insights Actions

16 High level IoT Architecture Devices RTOS, Linux, Windows, Android, ios Field Gateway Protocol Adaptation Protocol Adaptation Bi-directional device-cloud Telemetry ingestion Command & Control Device Registry & Identity Device Mgmt HTTP, AMQP, MQTT Cold Path Analytics Azure HDInsight, AzureML Azure Data Lake, Data Lake Analytics Hot Path Analytics Azure Stream Analytics Azure HDInsight Azure Time Series Insights Power BI App Service Web Apps Mobile Apps Field Gateway Cloud Gateway IoT Hub Hot Path Business Logic Service Fabric Azure Functions Logic Apps Notification Hubs BizTalk Services Device Connectivity & Management Analytics & Operationalized Insights Presentation & Business Connectivity

17 Lets focus on the stuff we already know Devices RTOS, Linux, Windows, Android, ios Field Gateway Protocol Adaptation Protocol Adaptation Bi-directional device-cloud Telemetry ingestion Command & Control Device Registry & Identity Device Mgmt HTTP, AMQP, MQTT Cold Path Analytics Azure HDInsight, AzureML Azure Data Lake, Data Lake Analytics Hot Path Analytics Azure Stream Analytics Azure HDInsight Azure Time Series Insights Power BI App Service Web Apps Mobile Apps Field Gateway Cloud Gateway IoT Hub Hot Path Business Logic Service Fabric Azure Functions Logic Apps Notification Hubs BizTalk Services Device Connectivity & Management Analytics & Operationalized Insights Presentation & Business Connectivity

18 Hot Path Analysis

19 Hot path analysis Azure Stream Analytics

20 Query Language Declarative queries in a SQL like language Easy to author and deploy Temporal semantics All operators respect, and some use, the temporal properties of events Built-in operators and functions Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation

21 SAQL Language & Library SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER/LEFT OUTER JOIN UNION CROSS/OUTER APPLY CAST INTO ORDER BY ASC, DSC WITH PARTITION B OVER DateName, DatePart, Day Month, Year, DateDiff DateTimeFromParts, DateAdd ABS, CEILING, EXP, FLOOR POWER, SIGN, SQUARE SQRT Lag IsFirst,Last CollectTop TumblingWindow HoppingWindow SlidingWindow SUM COUNT AVG MIN MAX STDEV STDEVP VAR VARP TopOne Len, Concat, CharIndex Substring, Lower Upper, PatIndex

22 Tell me the count of tweets per time zone every 10 seconds A 10-second Tumbling Window Time (secs) SELECT TimeZone, COUNT(*) AS Count FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY TimeZone, TumblingWindow(second,10)

23 A 10-second Hopping Window with a 5-second Hop Every 5 seconds give me the count of tweets over the last 10 seconds SELECT Topic, COUNT(*) AS TotalTweets FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY Topic, HoppingWindow(second, 10, 5)

24 A 10-second Sliding Window Give me the count of tweets for all topics which are tweeted more than 10 times in the last 10 seconds SELECT Topic, COUNT(*) FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY Topic, SlidingWindow(second, 10) HAVING COUNT(*) > 10

25 Hot path analysis Azure Stream Analytics Azure Time Series Insight

26 Azure Time Series Insights IoT scale time-series data store Schema-less store, just send data Easy IoT Hub connection Store, query and visualize billions of events Simple and fast navigation

27 Hot path analysis Azure Stream Analytics Azure Time Series Insight Azure HDInsight

28 Big Data in Azure IaaS Hadoop HDInsight Data Lake Analytics Hadoop Spark HBase Storm Hadoop distros on Azure VMs Azure-managed Hadoop clusters Big Data as a Service Blob Storage Data Lake Store Hyper-scale Storage optimized for analytics

29 Cold Path Analysis

30 Cold path analysis Azure Data Lake Store

31 Petabyte size files and Trillions of objects Azure Data Lake Store A No limits Data Lake that powers Big Data Analytics The first cloud Data Lake for enterprises that is secure, massively scalable and built to the open HDFS standard. With no limits to the size of data and the ability to run massively parallel analytics, you can now unlock value from all your unstructured, semi-structured and structured data. Scalable throughput for massively parallel analytics HDFS for the Cloud Always encrypted, Role-based Security & Auditing Enterprise-grade Support

32 Cold path analysis Azure Data Lake Store Azure Data Lake Analytics

33 Start in seconds, Scale instantly, Pay per job Azure Data Lake Analytics Massively parallel, extensible, analytics made simple The first cloud analytics service where you can easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python and.net over petabytes of data. With no infrastructure to manage, process data on demand, scale instantly, and only pay per job. Develop massively parallel programs with simplicity Debug and Optimize your Big Data programs with ease Virtualize your analytics Enterprise-grade Support and Security

34 DateTime = DateTime.Now; = EXTRACT OrderId int, Customer string, Date DateTime, Amount float FROM "/input/orders.txt" USING = SELECT * FROM orders WHERE Date >= startdate AND Date <= = SELECT * FROM orders WHERE Customer.Contains( Contoso ); TO "/output/output.txt" USING Outputters.Tsv(); U-SQL Example DECLARE constant values C# Expressions RowSets enable dataflow programming EXTRACT performs schema on read for files OUTPUT for writing files Built-in handling for CSV & TSV Whole-script optimization

35 REFERENCE ASSEMBLY [ExtPython]; def get_mentions(tweet): return ';'.join( ( w[1:] for w in tweet.split() if w[0]=='@' ) ) def usqlml_main(df): del df['time'] del df['author'] df['mentions'] = df.tweet.apply(get_mentions) del df['tweet'] return df = SELECT * FROM (VALUES ("D1","T1","A1","@foo Hello ("D2","T2","A2","@baz Hello ) AS D( date, time, author, tweet ); Python Extensions Use U-SQL to create a massively distributed program. Executing Python code across many nodes. Using standard libraries such as numpy and = ON date PRODUCE date string, mentions string USING new Extension.Python.Reducer(pyScript:@myScript);

36 Cold path analysis Azure Data Lake Store Azure Data Lake Analytics Azure Machine Learning

37 Azure Machine Learning AZURE MACHINE LEARNING SERVICES AZURE TRAIN & DEPLOY OPTIONS Spark SQL Server Virtual machines GPUs Container services Experimentation and Model Management Services Notebooks IDEs Azure Machine Learning Workbench VS Code Tools for AI ON- PREMISES EDGE SQL Server Machine Learning Server Azure IoT Edge

38 Cold path analysis Azure Data Lake Store Azure Data Lake Analytics Azure Machine Learning And good old SQL

39 {} T-SQL Java C/C++ C#/VB.NET PHP Node.js Python Ruby SQL Server The platform of choice Any data Any application Linux Anywhere Choice of platform

40 High level IoT Architecture Devices RTOS, Linux, Windows, Android, ios Field Gateway Protocol Adaptation Protocol Adaptation Bi-directional device-cloud Telemetry ingestion Command & Control Device Registry & Identity Device Mgmt HTTP, AMQP, MQTT Cold Path Analytics Azure HDInsight, AzureML Azure Data Lake, Data Lake Analytics Hot Path Analytics Azure Stream Analytics Azure HDInsight Azure Time Series Insights Power BI App Service Web Apps Mobile Apps Field Gateway Cloud Gateway IoT Hub Hot Path Business Logic Service Fabric Azure Functions Logic Apps Notification Hubs BizTalk Services Device Connectivity & Management Analytics & Operationalized Insights Presentation & Business Connectivity

41 But what good is it without data from devices Devices RTOS, Linux, Windows, Android, ios Field Gateway Protocol Adaptation Protocol Adaptation Bi-directional device-cloud Telemetry ingestion Command & Control Device Registry & Identity Device Mgmt HTTP, AMQP, MQTT Cold Path Analytics Azure HDInsight, AzureML Azure Data Lake, Data Lake Analytics Hot Path Analytics Azure Stream Analytics Azure HDInsight Azure Time Series Insights Power BI App Service Web Apps Mobile Apps Field Gateway Cloud Gateway IoT Hub Hot Path Business Logic Service Fabric Azure Functions Logic Apps Notification Hubs BizTalk Services Device Connectivity & Management Analytics & Operationalized Insights Presentation & Business Connectivity

42 Connect & Control Azure IoT Hub

43 Azure IoT Hub Device Management Device twin Software Firmware Configuration Device twin IoT Hub

44 Azure IoT Hub Device Management Device twin Desired Reported Methods IoT Hub Device twin Desired Reported Tags Methods Jobs Schedule and broadcast Device twin changes across large fleets

45 Azure IoT Hub Message Routes Remember before inbox rules? IoT Solution Devices Routing Logic Alarm Queue Event Processing Queue IoT Hub Storage Queue

46 Device Connectivity Device IoT Client Custom Protocols Custom Protocol Gateway (Cloud Service, VM) Device IoT Client AMQP, MQTT, HTTPS Device CoAP, AllJoyn, OPC Field Gateway* AMQP, MQTT, HTTPS IoT Hub Device IoT Client Data Path Device CoAP, AllJoyn, OPC Device Field Gateway OPC, HTTP, CoAP VPN/ExpressRoute Custom Protocol Gateway (Cloud Service, VM) Optional solution component Azure IoT solution component * Field Gateway is currently a concept that may represent Microsoft, 3rd party or custom capabilities at a hardware or software level

47 Industrial protocols Ethernet/IP Modbus OPC UA OPC DA Zigbee ProfiNet And many, many more Profibus BACNet KNX Source: en.wikipedia.org/wiki/list_of_automation_protocols

48 IoT Pattern Things Cloud Gateway Insights Actions

49 IoT Pattern + Edge Things Insights Actions Cloud Gateway Insights Actions

50 Azure Machine Learning TRAIN & DEPLOY OPTIONS AZURE MACHINE LEARNING SERVICES AZURE Spark SQL Server Virtual machines GPUs Container services Experimentation and Model Management ON-PREMISES SQL Server Machine Learning Server Notebooks IDEs Azure Machine Learning Workbench EDGE COMPUTING Azure IoT Edge

51 High level IoT Architecture Devices RTOS, Linux, Windows, Android, ios Field Gateway Protocol Adaptation Protocol Adaptation Bi-directional device-cloud Telemetry ingestion Command & Control Device Registry & Identity Device Mgmt HTTP, AMQP, MQTT Cold Path Analytics Azure HDInsight, AzureML Azure Data Lake, Data Lake Analytics Hot Path Analytics Azure Stream Analytics Azure HDInsight Azure Time Series Insights Power BI App Service Web Apps Mobile Apps Field Gateway Cloud Gateway IoT Hub Hot Path Business Logic Service Fabric Azure Functions Logic Apps Notification Hubs BizTalk Services Device Connectivity & Management Analytics & Operationalized Insights Presentation & Business Connectivity

52 Who (or what) are we building this for Devices RTOS, Linux, Windows, Android, ios Field Gateway Protocol Adaptation Protocol Adaptation Bi-directional device-cloud Telemetry ingestion Command & Control Device Registry & Identity Device Mgmt HTTP, AMQP, MQTT Cold Path Analytics Azure HDInsight, AzureML Azure Data Lake, Data Lake Analytics Hot Path Analytics Azure Stream Analytics Azure HDInsight Azure Time Series Insights Power BI App Service Web Apps Mobile Apps Field Gateway Cloud Gateway IoT Hub Hot Path Business Logic Service Fabric Azure Functions Logic Apps Notification Hubs BizTalk Services Device Connectivity & Management Analytics & Operationalized Insights Presentation & Business Connectivity

53

54 Questions?

55 Get started building IoT solutions today 50 Announced Azure regions world wide Hyper-Scale Capacity 14 Azure IoT Suite regions AZURE REGIONS AZURE IOT SUITE REGIONS

56 Thank you for your time

57