Oliver Engels & Tillmann Eitelberg. Big Data! Big Quality?

Transcription

1 Oliver Engels & Tillmann Eitelberg Big Data! Big Quality?

2 Like to visit Germany? PASS Camp 2017 Main Camp (4.12 Kick Off Evening) Lufthansa Training & Conference Center, Seeheim SQL Konferenz 2018 PreCon: MainCon: Darmstadmium, Darmstadt

3

4 Big Data Big Quality?

5 DQM Theory: What is Data Quality? Data Quality represents the degree to which the data is suitable for business usages Data Quality is built through People + Technology + Processes Bad Bata = Bad Business

6

7 DQM Theory: Cost of Data Quality The costs of bad data quality are most of the time hidden or implicite accepted Data quality cost Costs because of bad data quality Cost of optimizing data quality Direct Indirect Prevention Discovery Cleansing

8 DQM Theory: Cost of Data Quality Cost Costs due to bad data quality Costs to avoid or correct bad data quality 80% Quality 100%

9 DQM Theory: Reasons for bad data quality Data Capturing Expiration Usage Data Quality Processes Architecture Definition

10 DQM Theory: Tasks to solve bad Data Quality Standard Workflow in Data Quality Optimization Validation & Profiling

11 Big Data Big Quality?

12 No! Even more problems!

13 PAC 2012

14 Big Data Even more problems! Volume: Increase of volume More to process Velocity: Realtime processing concepts for quality check and improvement Variety: Variety = More complexity for quality processes

15 Big Data Even more problems! Veracity: The accuracy of data is in doubt by default in Big Data Value: New business models relying on Big Data exponentially create the financial impact for data quality problems

16 Big Data - Tasks to solve bad Data Quality

17 Findings Bad Data Bad Business Data Quality is still a problem in most cases underestimated or hidden Big Data = The 5 V s = Even more problems!

18 OK! Let s see, if we have possibilities to solve problems

19 Check What has the MS Data Platform to offer?

20 Modern Data Platform Landscape Cloud born data Hybrid & Hyperscale Data Platform Landscape External Data File, WebServices (e.g. Market Research) IaaS Data (e.g. Hosted Custom Application) Presentation Layer Visualization Layer Visual Output PaaS Data (e.g. Custom Application) Governance Layer SaaS Data (e.g. CRM, Logistics) Analytical Layer Analytical Output Streaming Data (e.g. Sensor, Devices) Unstructured / Semi-structured (e.g. Video, Image) Storage Layer Ingest Layer Integration Layer Raw Output On Prem born data Legacy systems (e.g. ERP, CRM) Relational and Multi Dimensional Data (e.g. Custom App) Document based data (e.g. XLS, DOC, PPT, PDF, CSV) Document based data (e.g. XLS, DOC, PPT, PDF, CSV)

21 Modern Data Platform Landscape # Layer Function 1 Ingest Ingestion of data to a centralized storage environment (ETL, ELT, Raw data pump, Replication) 2 Storage Data Lake with unlimited data capacity and storage of structured, semi-structured and unstructured data 3 Integration Integration of Meta-, Master data as well as Data Quality, Profiling and Lineage 4 Analytical Relational, Multidimensional, Predictive, Stream analytical compute systems with distributed or in-memory capabilities 5 Governance Data Governance and Data Catalog incl. KPI portfolio, Corporate Identify, Security 6 Visualization Explorational data discovery and standard reporting 7 Presentation Any device & multi platform presentation layer

22 Modern Data Platform Landscape Portal Integration App Integration Search Integration Mobile Client Broadcast and Subscriber Pixel Perfect Visualization Explorative Visualization Statistic Visualization (R) Spatial Visualization Ad-hoc Functional Mapping Data Governance Record Management Performance Management Compliance Visual Master Data Text and Media Analytics Cognitive Analytics Predictive Analytics Azure Stream Analytics Transactional Analytics Multidimensional Analytics Statistic Stream Analytics Federated Search Master Data Management Meta Data Data Lineage Data Profiling Relational Storage Semi Structured Data Storage Unstructured Data Storage Data Lake ETL and ELT Replication and Sync Raw Data Pump

23 On Prem Microsoft Data Platform Landscape Cloud (Hyperscale) SharePoint SQL Server SSRS Portal Power BI Mobile Client Office 365 SharePoint Azure Apps R-Visual Studio or R-Studio Power BI Desktop (+R) SQL Server Reporting Services (+R) Excel Power BI Service Power BI Embedded SQL Server Master Data Services SQL Server Data Quality Services Azure Data Catalog Power BI Content Packs Power Apps Common Data Services SQL Server Polybase Engine SQL Server R-Server Engine Azure Machine Learning Azure HDInsight Spark/Hadoop Azure Stream Analytics / Storm SQL Server Operational Analytics Engine SQL Server Multidimensional Tabular Engine Azure SQL Database Azure SQL Data Warehouse Azure Data Lake Analytics (U-SQL) SQL Server Master Data Services SQL Server Data Quality Services Azure Data Catalog Power Apps Common Data Services SQL Server Relational Engine File Storage Azure SQL DB & Data Warehouse Azure Data Lake Azure BLOB Storage SQL Server Integration Services Azure Data Factory Azure Logic Apps

24 Microsoft Data Platform Landscape: Tools you can use for DQ by Layer MS Data Platform Service SQL Server Integration Service (SSIS) Azure Logic Apps SQL Server Master Data Services SQL Server Data Quality Services Azure Data Catalog Power Apps Common Data Services Layer Ingest Ingest Integration & Governance Integration & Governance Integration & Governance Integration & Governance

25 Microsoft Data Platform Landscape: Tools you can use by DQ Function MS Data Platform Service SQL Server Integration Service (SSIS) x x x x x x Azure Logic Apps (x) x SQL Server Master Data Services x x x x SQL Server Data Quality Services x x x x x Azure Data Catalog x x x Power Apps Common Data Services x x x Azure Data Lake Analytics x x x x SQL Server R / P Integration x x x x x x Azure HDInsight Spark (R) x x x x x x

26 NOW! Let s look into some solutions in detail!

27 Let s take these:

28 SQL Server R Language integration

29 This is R! R is an interpreted language Mostly R is used through a command-line interpreter Like other scientific languages e.g. MATLAB, R supports matrix arithmetic R uses procedural programming with functions In some cases object-oriented programming is possible The capabilities of R are extended through user-created packages, which allow specialized statistical techniques, graphical devices, import/export capabilities etc These packages are developed primarily in R A core set of packages is included with the installation of R Nearly 8000 (May 2016) packages are available via the Comprehensive R Archive Network (CRAN)

30 Microsoft is now a R company

31 This is R! R has a lot of libraries that also include functions you can use for data quality e.g.: summary distribution histograms regressions dataqualityr or datacheck These libraries / packages are not always per default for DQ purpose, but they enable you to build a nice framework for functions to monitor or improve DQ.

32 SQL Server Master Data Services

33 MDS: Main features from a data quality perspective - Versioning - Archiving / historical background - Applications can still work with historic versions of the data if necessary - Safe-keeping of validated versions - Transactions / Change Sets - Log and manage changes - Central Approver can either approve and commit or rollback changes - Business Rules - Check data for validity while entering data - Only valid versions can be commited - Define rules by pre-defined editor or define logic with User functions in T- SQL

34 DEMO!

35 Azure Data Catalog Data Repository

36 What is Azure Data Catalog? An enterprise-wide catalog in Azure that enables self-service discovery of data from any source A metadata repository that allow users to register, enrich, understand, discover, and consume data sources

37 Azure Data Catalog is part of Cortana Intelligence

38 What can Azure Data Catalog do for DQ?

39 What can Azure Data Catalog do for DQ? Publisher Consumer IT Admin Publish Enric h Discove r Register Data Sources Browse - Search Categorize Annotate Understan d Get context Identify Intent Extend Govern Apply Policies - Control Access Analyz e Track and monitor usage Use open REST APIs to integrate data publishing and discovery into existing applications and processes

40 What can Azure Data Catalog do for DQ? Programmability Scenarios Incorporating data source discovery into client applications Future: Defining custom data source and data asset types ( type extensibility ) REST APIs Register data asset operations: Register Data Asset, Delete Data Asset, and Search Data Asset Annotate data asset operations: Annotate Data Asset, Update Annotation, Get Data Asset with Annotations, and Delete Annotation Search API uses familiar, powerful, search syntax

41 What can Azure Data Catalog do for DQ?

42 What can Azure Data Catalog do for DQ? Search Browse Filter Discover Understand Analyze Metadata Experts Context Your data Your tools Your way Consume Contribute Tag Document Publish

43 DEMO!

44 What can Azure Data Catalog do for DQ? Register data sources Registration stores key information (metadata) about the data source such as names, types of data, and location of data in the catalog. Only information about the data source is stored in Data Catalog. Use the Data Source Registration tool to register data sources.

45 What can Azure Data Catalog do for DQ? Data Catalog has multiple ways to discover data assets, including Simple keyword search Interactive filters An advanced search syntax for power users.

46 What can Azure Data Catalog do for DQ? Annotations enhance information about data sources so that they are easier to discover and understand. Any Data Catalog user can provide annotations, so it s easy for every user with a perspective on the data to share it. Annotation Expert Annotation

47 Power Apps Common Data Service

48 Common Data Services: What they are intended to do! A library of entities that represent the most commonly used entities across business and productivity applications The library is organized for simplicity and rapid discoverability Typical solutions will revolve around core elements of enterprise data for example, an employee, a customer, a product, or a supplier Applications across the enterprise will drive updates to that data The model is implemented to simplify the challenges of data management.

49 Common Data Services: What they are intended to do! Connect to Common Data Model or existing systems and data sources Build apps, forms, and workflows without writing code Publish apps instantly for web and mobile

50 Common Data Services: What they are intended to do! Build modern business apps fast Compose analytics, user experiences, and automation using an integrated set of services from Microsoft Empower line of business Power users who understand their business needs best can now build solutions quickly without writing code Enable developers to extend Seamlessly provide capabilities and components for power users leveraging Azure platform services Azure Data analytics SQL Server BI Azure AppServices Azure Functions Azure Logic Apps Azure API Management Connect to data across apps and platforms Whether built on the Common Data Model or integrated with existing systems and services, get more value from your data

51 Common Data Services: What they are intended to do! Backbone for all business data

52 Common Data Services: What they are intended to do! User Security: Users who can use apps and flows that are shared with them Creators who can create and edit apps and flows Administrators who manage environments including data security Database Owners who can access all data and change entity schema

53 Common Data Services: What they can do for DQ Entity Model with Attributes, Relationships? Like MDS = Master Data Capabilities Centralized Services that can be connected via Flow and Logic Apps from all data sources on prem and in the cloud = Data Hub = Enrichment and Meta Data Management Can be directly used by Power BI as a Model = Standardization, High Quality Datasource

54 DEMO!

55 Findings Bad Data Bad Business Not all is DQS Microsoft offers a variety of tools to solve DQ Problems A lot!= well integrated A real suite or workbench is completely missing R or Python Integration is helpful Common Data Services should be considered MDS, DQS and SSIS have still their existence! Microsoft should take DQ more seriously = Strategy