Take P, R or U. and solve your data quality problems Oliver Engels & Tillmann Eitelberg, OH22

Transcription

1 Take P, R or U and solve your data quality problems Oliver Engels & Tillmann Eitelberg, OH22

2 Oliver Engels CEO, oh22data Datamonster from Germany MS Data Platform MVP President of PASS Germany e.v. Regional Mentor Germany SQL Saturday / SQL Konferenz Organizer Chapter Leader Frankfurt Microsoft ptsp Tillmann Eitelberg CEO oh22information Datamonster from Germany MS Data Platform MVP Vice-President of PASS Germany e.v. Regional Mentor Germany SQL Saturday / SQL Konferenz Organizer Chapter Leader Rheinland

3 Our topic today: OMG! BAD DATA!

4

5 What is Data Quality? Data Quality represents the degree to which the data is suitable for business usages Data Quality is built through: People + Technology + Processes Bad Bata = Bad Business

6 Cost of Data Quality The costs of bad data quality are most of the time hidden or implicite accepted Data quality cost Costs because of bad data quality Cost of optimizing data quality Direct Indirect Prevention Discovery Cleansing

7 Cost of Data Quality Cost Costs due to bad data quality Costs to avoid or correct bad data quality 80% Quality 100%

8 Reasons for bad data quality Data Capturing Expiration Usage Data Quality Processes Architecture Definition

9 Workflow in Data Quality Optimization Validate & Profile

10 DQ in Modern Data Platform landscapes Cloud born data Hybrid & Hyperscale Data Platform Landscape External Data File, WebServices (e.g. Market Research) IaaS Data (e.g. Hosted Custom Application) Presentation Layer Visualization Layer Visual Output PaaS Data (e.g. Custom Application) Governance Layer SaaS Data (e.g. CRM, Logistics) Analytical Layer Analytical Output Streaming Data (e.g. Sensor, Devices) Unstructured / Semi-structured (e.g. Video, Image) Storage Layer Ingest Layer Integration Layer Raw Output On Prem born data Legacy systems (e.g. ERP, CRM) Relational and Multi Dimensional Data (e.g. Custom App) Document based data (e.g. XLS, DOC, PPT, PDF, CSV) Document based data (e.g. XLS, DOC, PPT, PDF, CSV)

11 DQ in Modern Data Platform landscapes # Layer Function 1 Ingest Ingestion of data to a centralized storage environment (ETL, ELT, Raw data pump, Replication) 2 Storage Data Lake with unlimited data capacity and storage of structured, semi-structured and unstructured data 3 Integration Integration of Meta-, Master data as well as Data Quality, Profiling and Lineage 4 Analytical Relational, Multidimensional, Predictive, Stream analytical compute systems with distributed or in-memory capabilities 5 Governance Data Governance and Data Catalog incl. KPI portfolio, Corporate Identify, Security 6 Visualization Explorational data discovery and standard reporting 7 Presentation Any device & multi platform presentation layer

12 DQ in Modern Data Platform landscapes Portal Integration App Integration Search Integration Mobile Client Broadcast and Subscriber Pixel Perfect Visualization Explorative Visualization Statistic Visualization (R) Spatial Visualization Ad-hoc Functional Mapping Data Governance Record Management Performance Management Compliance Visual Master Data Text and Media Analytics Cognitive Analytics Predictive Analytics Azure Stream Analytics Transactional Analytics Multidimensional Analytics Statistic Stream Analytics Federated Search Master Data Management Meta Data Data Lineage Data Profiling Relational Storage Semi Structured Data Storage Unstructured Data Storage Data Lake ETL and ELT Replication and Sync Raw Data Pump

13 MS Mapping On Prem Cloud (Hyperscale) SharePoint SSRS PBI Server Power BI Mobile Client Office 365 SharePoint Azure Apps R-Visual Studio or R-Studio Power BI Desktop (+R) SQL Server Reporting Services (+R) Excel Power BI Service Power BI Embedded SQL Server Master Data Services SQL Server Data Quality Services Azure Data Catalog Power BI Content Packs Power Apps Common Data Services SQL Server Polybase Engine SQL Server ML Engine Azure Machine Learning Azure HDInsight Spark/Hadoop Azure Stream Analytics / Storm SQL Server Operational Analytics Engine SQL Server Multidimensional Tabular Engine Azure SQL Database & DWH Azure Analysis Service Azure Data Lake Analytics (U-SQL) SQL Server Master Data Services SQL Server Data Quality Services Azure Data Catalog Power Apps Common Data Services SQL Server Relational Engine File Storage Azure SQL DB & Data Warehouse Azure Data Lake Azure BLOB Storage SQL Server Integration Services Azure Data Factory Azure Logic Apps

14 Hmm! Data Quality Services (DQS) Microsofts Solution for Data Quality

15 DQS (Data Quality Services) Great idea to integrate DQ functions in the MS Data Platform BUT: Since 2012 no real further development Product still a version 1.0 with small depth in functionality, performance and UI issues No roadmap available Usable for small projects, but not for bigger things! Any help?

16 DQ Solutions in the MS Data Platform In this session we will introduce you to three magic letters of scripting languages, you can use for Data Quality in the MS Data Platform: P (Python), R and U (U-SQL) can be greatly leveraged to help you with your DQ problems and we show you in our samples how

17 Script languages we use for DQ Python is a widely used high-level programming interpreted language for general-purpose programming, It emphasizes code readability and implementation in small and large scale applications R is an open source programming interpreted language and software environment for statistical computing and graphics. The R language is widely used among statisticians and data miners for developing statistical software and data analysis U-SQL is a query language designed for Azure Data Lake. It provides a way to mingle SQL keywords with syntactic C# expressions, so that within a single script, a programmer can schematize the data from an unstructured source, use SQL to aggregate the data into the desired form (Wikipedia) (Stackoverflow)

18 Applying P, R & U to the DQ workflow You will see the following samples as a subset of possibilities with these languages DQ Scoring DQ KB DQ Dedub

19 Using R as DQ Tool Using R Libraries Develop in Visual Studio Operationalize in SQL Server and Spark Demo: Profile & Scoring Analysis with R

20 DEMO R for Data Quality

21 Recap

22 Using P as DQ Tool Using Python Modules Develop in Visual Studio / Rodeo / Jupyter Operationalize in SQL Server and Spark Demo: Knowledge Base Techniques with P Deduplication Algorithms used in P

23 Knowledge Base Matches > 90 % Cleaned Data Source Data Knowledge Base (Synonyms) Matches > 80 % Knowledge Base (e.g. MDS, SharePoint) Data Steward Manual Validation

24 DEMO P for Data Quality

25 Using ADLA (U-SQL) as DQ Tool Using U-SQL with C# Assemblies Develop in Visual Studio Operationalize in Azure Data Analytics Demo: Deduplication with U-SQL Data Enrichment through Federation

26 Using ADLA (U-SQL) as DQ Tool Duplicates classification Is Duplicate Non-Duplicate Found as Duplicate Not found as Duplicate

27 DEMO U for Data Quality

28 Data Quality Dashboard with PBI on ADL

29 DQ with P, R and U > Findings Advantages Disadvantages Great variety of libraries and modules for DQ in P and R, good extensibility of U through assemblies Flexibility to adjust to your companies DQ problems A lot of possibilities to integrate into current data landscapes on prem and in the cloud Ability to scale Not for non technical users No GUI available Not a service Framework with different libraries and versions difficult to know what is useful

30 Enjoy the SQL Server Konferenz!