NYC Cloud Machine Learning Meetup. Introduction to Cortana Analytics

Transcription

1 NYC Cloud Machine Learning Meetup Dec 1, 2015 Introduction to Cortana Analytics note: deck will be posted to the meetup site more->files tab.

2 Topics What is Cortana Analytics What are the core components What is there besides machine learning How can I start using these components Some pointers and references

3 ?

4 Cortana Analytics Suite Transform data into intelligent action Business apps People Custom apps Sensors and devices DATA INTELLIGENCE Automated Systems ACTION

5 FULLY MANAGED ALL TYPES OPERATIONALIZE SHARE AND COLLABORATE OPENNESS END-TO-END

6 Demo Architecture Diagram-Connected Cars DATA SOURCES INGEST PREPARE ANALYZE PUBLISH CONSUME Diagnos7c streaming Machine Learning Event hubs Stream Analy7cs Cortana Data in mo;on Stream Analy7cs Data at rest HDInsight HDInsight Machine Learning PowerBI Business apps Azure DataLake Azure SQL Data Warehouse Vehicle Catalog Data Factory: Move Data, Orchestrate, Schedule, and Monitor Data Catalog: Register, Annotate, Understand, Discover Data Sets

7

8 Data Sources Ingest Prepare (Normalize, Clean) Analyze (Build Dimension Model, ML) Publish (Prog. consumption, BI/visualization) Consume ADF Pipeline ADF Pipeline Web Log Files batch ADF Pipeline Blob Storage ADF Pipeline HDI HDI, Custom Code HDI, Machine Learning Azure Blob Storage Relational DW/DM Apps Retailer Website - Data sources are on-prem, SaaS or cloud - Volume per move ~100GB or lower - Most commonly collected nightly or hourly - Data is staged - Processing step: scrub for compliance purposes, partition for long term storage - HDI & Customer code used in this step as a transformation/cleaning tool - E.g. enrich, normalize, etc. - Generate BI-Ready results (e.g. dims or facts, aggregated big data, etc.) - Create result set to drive app or business process (e.g. product recommendations list for displaying on a website) - May be on-prem or cloud - Create result set to drive app or business process (e.g. product recommendations list for displaying on a website) - Apps here means any programmatic consumption. Most common cases seen to date are backend of web apps

9 HDInsight

10

11

12 @searchlog = EXTRACT UserId int, Start DateTime, Region string, Query string, Duration int?, Urls string, ClickedUrls string FROM "/Samples/Data/SearchLog.tsv" USING Extractors.Tsv(); TO "/output/searchlog-first-u-sql.csv" USING Outputters.Csv();.

13

14

15

16

17

18

19

20 Adding magic to your business with Perceptual Intelligence Chris Hewitt ARC323B

21 People are better than machines At communicating and understanding: But People are expensive And can t handle huge volumes of communications ( s, tweets, posts, phone calls)

22 Abc Vision Face Emotion Speech LUIS Spell Text

23

24

25

26

27

28

29

30

31 abc

32 abc This presentation is very interesting. The Text Analytics API's pre trained model makes sentiment analysis easy.

33

34

35 R usage continues to grow

36 Revolution R vs open source R Open source R exceeds RAM and fails RRE scales linearly well beyond RAM limits As data grows RRE optimization becomes apparent Compressed File Size (MB) Open Source R (secs) Revolu;on R (secs) File Name No. Rows Tiny 0.3 1, V. Small , Small , Medium ,235, Large ,353, Big (full) 12,960.0 V.Big 25, ,534,96 9 Memory! ,069,93 8 Memory! ,139,87 6 Memory! Public US Flight Data Huge Linear Regression 51,840.2 on Arrival Delay Run on 4 core laptop, 16GB RAM and 500GB SSD

37 The Revo product suite Revolution R Open Free and open source R distribution Enhanced and distributed by Revolution Analytics Revolution R Enterprise Secure, Scalable and Supported Distribution of R With proprietary components created by Revolution Analytics

38 RRE: scaling to big data mpg cyl disp hp drat wt qsec vs am gear carb In Hadoop Slave node Master node Slave node Slave node

39 Cortana Analytics Process Questions or Feedback NYC Taxi Data

40 Code Session Name Start Date and Time End Date and Time Presenter(s) Venue/Room Level DAT211 Keynote : Data Platform and Insights Track - by Ulrich Homann, Distinguished Architect for Azure 17/11/ :30 17/11/ :45 Ulrich Homann Arena 2 Level 200 DAT212 SQL Server Evolution, SQL 2016 innovations 17/11/ :00 17/11/ :15 Cameron Gardiner Arena 2 Level 200 DAT213 Using the Cortana Analytics Process - An In-depth Walkthrough 17/11/ :45 17/11/ :00 Barnam Bora Room 5 & 6 Level 200 DAT321 Microsoft Azure SQL Data Warehouse Overview 18/11/2015 8:15 18/11/2015 9:30 Matthew Winter, John Hoang Central C Level 300 DAT323 Apache Spark for Azure HDInsight Overview 18/11/ :30 18/11/ :45 Matthew Winter Room 5 & 6 Level 300 DAT324 SQL Server 2016 Scale and diagnostics engine enhancements 18/11/ :45 18/11/ :00 Mike Weiner Central B Level 300 DAT325 What's new in SQL Server 2016 Integration Services (IS) and Master Data Services (MDS) 18/11/ :30 18/11/ :45 Jimmy Wong Room 5 & 6 Level 300 DAT226 Temporal, Query store and JSON support in SQL Server /11/ :00 18/11/ :15 Mike Weiner, John Hoang Room 7 & 8 Level 200 DAT331 Securing Data on your terms 19/11/2015 8:15 19/11/2015 9:30 Kristina Rumpff Arena 1A Level 300 DAT332 NoSQL PaaS in Azure through DocumentDB - The Real Deal 19/11/2015 9:45 19/11/ :00 Travis Sansome Room 7 Level 300 DAT333 Working with SQL Server Spatial Data 19/11/ :30 19/11/ :45 Greg Low Central C Level 300 DAT334 Connected Cars, Product Recommendations and More - A deep dive into Azure Data Factory and the Cortana Analytics Suite 19/11/ :45 19/11/ :00 Gaurav Malhotra Arena 2 Level 300 DAT335 All about Revolution R Enterprise 19/11/ :30 19/11/ :45 Hong Ooi Room 9 Level 300 DAT335 Being a DBA is cool again with SQL /11/ :30 19/11/ :45 Matt Lavery, Joanna Podgoetsky Central C Level 300 DAT336 How Available is SQL Server /11/ :00 19/11/ :15 Warwick Rudd, Henry Rooney Room 5 & 6 Level 300 DAT341 Hybrid data platform â making the most of Azure plus your on-prem kit 20/11/2015 8:30 20/11/2015 9:45 Martin Cairney Room 9 Level 300 DAT341 Power BI : The developer edition 20/11/2015 8:30 20/11/2015 9:45 Travis Sansome Central A Level 300 DAT342 Realizing the Benefits of Upgrading Microsoft SQL Server 20/11/ :15 20/11/ :30 Victor Isakov Room 8 Level 300 DAT442 Using Live Query Statistics to see the inner workings of your queries in SQL 2014 and /11/ :15 20/11/ :30 Rob Farley Room 9 Level 400 DAT343 Real world Power BI techniques 20/11/ :45 20/11/ :00 Grant Paisley Central C Level 300

41 Continue your Ignite learning path Visit Microsoft Virtual Academy for free online training visit Visit Channel 9 to access a wide range of Microsoft training and event recordings Head to the TechNet Eval Centre to download trials of the latest Microsoft products