The Cortana Intelligence Suite

Size: px

Start display at page:

Download "The Cortana Intelligence Suite"

Alfred Porter
5 years ago
Views:

1 Slide 1 The Cortana Intelligence Suite Foundations Data Discovery and Ingestion Microsoft Machine Learning and Data Science Team CortanaIntelligence.com Main page: To begin this module, you should have: Basic Math and Stats skills Business and Domain Awareness General Computing Background NOTE: These workbooks contain many resources to lead you through the course, and provide a rich set of references that you can use to learn much more about these topics. If the links do not resolve properly, type the link address in manually in your web browser. If the links have changed or been removed, simply enter the title of the link in a web search engine to find the new location or a corollary reference.

Slide 2 Section 2 Learning Objectives 1. Understand how to source and document data locations 2. Understand feature selection 3. Understand Azure Storage Options 4.

2 Slide 2 Section 2 Learning Objectives 1. Understand how to source and document data locations 2. Understand feature selection 3. Understand Azure Storage Options 4. Use various methods to ingest data into Azure Storage 5. Examine data stored in Azure Storage 6. Use various tools to explore data At the end of this Module, you will: Understand how to source and document data locations Understand feature selection Understand Azure Storage Options Use various methods to ingest data into Azure Storage Examine data stored in Azure Storage Use various tools to explore data

3 Slide 3 The Data Science Process and Platform

4 Slide 4 The Team Data Science Process Business Understanding Data Acquisition and Understanding Define Objectives Identify Data Sources Ingest Data Explore Data Update Data Modeling Feature Selection Create and Train Model Deployment Operationalize Customer Acceptance Testing and Validation Handoff Re-train and re-score This process largely follows the CRISP-DM model: It also references the Cortana Intelligence process: A complete process diagram is here: Some walkthrough s of the various services: An integrated process and toolset allows for a more close-to-intent deployment Iterations are required to close in on the solution but are harder tio management and monitor

5 Slide 5 The Cortana Intelligence Platform Cortana, Cognitive Services, Bot Framework Power BI Stream Analytics HDInsight Azure Machine Learning (MRS) SQL Data Warehouse (SQL DB, Document DB) Data Lake Event Hubs Data Factory Data Catalog Microsoft Azure Platform and Storage: Microsoft Azure Storage: (Host It) Azure Data Catalog: (Doc It) Azure Data Factory: (Move It) Azure Event Hubs: (Bring It) Azure Data Lake: (Store It) Azure DocumentDB: Azure SQL Data Warehouse: (Relate It) Azure Machine Learning: (Learn It) Azure HDInsight: (Scale It) Azure Stream Analytics: (Stream It) Power BI: (See It) Cortana: and and (Say It) Cognitive Services: Bot Framework:

6 All of the components within the suite: What can I do with it? Getting Started Quickly:

7 Slide 6 6 Data Validation:

Slide 7 In reference to machine learning, but applicable to all data usage: https://azure.microsoft.

8 Slide 7 In reference to machine learning, but applicable to all data usage: Great whitepaper on preprocessing data in R and important questions to ask: 382/images/EN-CNTNT-Whitepaper-Data-Prep-Using-R.pdf

9 Slide 8 Azure Data Catalog Full example:

10 Slide 9 Adding and searching data on the Azure Data Catalog Search for one on-line table involving your business scenario Connect to the Azure Data Catalog Add a Data Source as an HTTP site Add metadata to the information Save and view in Portal Search for your data element based on name or tag you added Add more tags Add yourself as an expert Search for your name as an expert

11 Slide Data Storage Options (Building Real-World Cloud Apps with Azure): Options: 4/big-data-and-azure/

12 Slide 11 Unstructured data such as media files, logs, binary data, backups Metadata (e.g. user info), in key-value format, fast and easy to query Messaging between components of your application Shared file systems option when your application is already built to use a SMB protocol { name : Sue, role : 'admin, status : active, location : WA Blob Table Queue File Storage-5-Minute-Overview

13 Slide 12 Redundancy and Location LRS: 3 Copies, 1 Datacenter GRS: 6 Copies, 2 Datacenters ZRS: 3 Copies, 2-3 Datacenters Locations and Redundancy Overview: Affects on Scalability and Performance Targets: Pricing Details:

14 Slide 13 Azure Portal - Azure PowerShell - AZCOPY - Azure CLI - Service management REST API - Azure Storage Resource Provider REST API -

15 Slide 14 Copy and view data on your storage account using AZCOPY Open the Azure Portal, locate your Storage Account (or create one if you have not), and a Container (or create one if you have not). Note the name of the SA and the Container, and your storage key. From your DVSM, or if you installed the Azure PowerShell tools locally, open a command prompt. If you do not have the AZCOPY command, download and install it here: Navigate to this page: Locate the section marked Blob: Upload - Upload single file and follow the instructions to load one file to your storage account, using your Storage Account and storage keys. Next, locate the section on the web page with instructions marked Blob: Download - Download single blob. Follow the instructions there to copy your file to a new folder on your local computer.

17 Slide Example of a 3 rd Party Solution:

18 Slide 16 PowerShell in Azure Storage - Azure Data Factory data movement - Azure Automation - Azure storage SDKs for examples see Azure tools and SDKs in general can be downloaded here - MS Azure Storage Explorer - AzCopy -

19 Import/Export service -

20 Slide

21 Slide Understanding the statistics of exploring data:

22 Slide 19 MRS Azure ML HDInsight Other Tools Data Exploration and Predictive Modeling with R - Data Exploration with Azure ML - Statistics Using Excel Sed, awk, grep (in Windows as well) -

$csv files in your \Resources folder, use R, Excel, Azure ML or any other exploration tools you ve seen in the class to explore the shape, size, layout, distribution$

23 Slide 20 Exploring your data Using the building.csv and HVAC.csv files in your \Resources folder, use R, Excel, Azure ML or any other exploration tools you ve seen in the class to explore the shape, size, layout, distribution and other characteristics you can find in the data. Document that in any format and be ready to discuss.

24 Slide Understand how to source and document data locations 2. Understand Azure Storage Options 3. Use various methods to ingest data into Azure Storage 4. Examine data stored in Azure Storage 5. Use various tools to explore data Questions?

White Paper / Azure Data Platform: Ingest

White Paper / Azure Data Platform: Ingest Contents White Paper / Azure Data Platform: Ingest... 1 Versioning... 2 Meta Data... 2 Foreword... 3 Prerequisites... 3 Azure Data Platform... 4 Flowchart Guidance...