Como liberar Insights de Negócio garantindo a Qualidade dos seus Dados? ÉRIKA MADEIRA

Size: px

Start display at page:

Download "Como liberar Insights de Negócio garantindo a Qualidade dos seus Dados? ÉRIKA MADEIRA"

Roger Watson
5 years ago
Views:

1 DAMA & MICROSOFT Como liberar Insights de Negócio garantindo a Qualidade dos seus Dados? ÉRIKA MADEIRA V-ERMADE@MICROSOFT.COM

2 2

3 Top 3 impediments Source: Information Week Reports,

4 Top Barrier for BI Source: Information Week Reports,

5 Data Quality Standard Issue Are data elements consistently defined and understood? Sample Data Problem Gender code = M, F, U in one system and Gender code = 0, 1, 2 in another system Complete Is all necessary data present? 20% of customers last name is blank, 50% of zip-codes are Accurate Valid Does the data accurately represent reality or a verifiable source? Do data values fall within acceptable ranges? A Supplier is listed as Active but went out of business six years ago Salary values should be between 60, ,000 Unique Data appears several times Both John Ryan and Jack Ryan appear in the system are they the same person? 5

6 DQ Issues and DQ Dimensions Name Gender Street House # Zip code City State D.O.B Before After John Doe Male 60th street 45 New York New York 08/12/64 Jane Doe Male Jonathan ln Poughkeepsy NY 21-dec-1954 Name Gender Street House # Zip City State D.O.B code John Doe Male E 60th St 45W New York NY 08/12/64 Jane Doe Female Jonathan Lane Poughkeepsie NY 12/21/54 Completeness Accuracy Conformity Consistency Uniqueness Before After Name Address Postal Code City State John Smith 545 S Valley View Drive # Anytown New York Margaret & John smith 545 Valley View ave unit Anytown New York Maggie Smith 545 S Valley View Dr Anytown New York John Smith 545 Valley Drive St NY NY Name Address Zip Code City State Cluster John Smith 545 S Valley View Drive # Anytown New York 1 Margaret & John smith 545 Valley View ave unit Anytown New York 1 Maggie Smith 545 S Valley View Dr Anytown New York 1 John Smith 545 Valley Drive St NY NY 2 6

7 MICROSOFT INTRODUCES DQS

8 XML KPIs SQL Server 2000 Management Studio Espelhamento SQL Server 2005 SQL Server Compactação Gerenciamento baseado em políticas Programação 2008 PowerPivot Integração com o SharePoint Master Data Services SQL Server 2008 R2 AlwaysOn Índice ColumnStore Data Quality Services Power View Conectividade de Nuvem SQL Server 2012 SQL Server In-Memory em todas as cargas de trabalho Desempenho e Escala Nuvem Híbrida Otimizada HDInsight BI de Nuvem 2014

of your data Acquires additional knowledge

9 Knowledge-Driven Semantics Knowledge Discovery Open and Extendible Easy to use Based on a Data Quality Knowledge Base (DQKB) Data Domains capture the semantics of your data Acquires additional knowledge the more you use it Add user-generated knowledge & 3 rd party reference data providers User experience designed for increased productivity

DQS Blog Tips, tricks and guidance on best practices for

com/b/dqs DQS Movies A set of getting started movies for

com/ enus/dn912438 DQS Forum https://social.m sdn.

10 DQS Blog Tips, tricks and guidance on best practices for using DQS courtesy of the DQS team blogs.msdn.com/b/dqs DQS Movies A set of getting started movies for an easy introduction to DQS microsoft.com/ enus/dn DQS Forum sdn.microsoft.c om/forums/sqls erver/enus/home?forum =sqldataquality services

11 The Data Platform world has changed! Volume, Variety, Velocity, Complexity Real time data Multiple data types Huge Data Volume Mobility User Expectative Hardware and Storage costs 11

12 HIGH VOLUMES : AMIL DataWarehouse with high performance AMIL IN NUMBERS: The biggest Healthcare Company in Brazil, and part of United Health Group (UHG) Almost 7 million customers for medical/dental insurance plans Almost 200,000 companies that buy their corporate plans with AMIL +25 million calls/year through its call center +60 million clinical tests being executed per year +26 million medical/dental visits/year +550,000 hospitalizations in 2012

13 HIGH VOLUMES : AMIL DataWarehouse with high performance Next Steps Current Situation 2 APS More than 1 thousand single users today More than 200 Reports/Dashboards Future Big Data Create a Hadoop Cluster in APS

14 High performance and rich visualizations at BI process

15 Delivering on one of the old dreams of Microsoft co-founder Bill Gates: Computers that can see, hear and understand. John Platt Distinguished scientist at Microsoft Research What is Machine Learning? Predictive computing systems become smarter with experience

16 The United States Postal Service processed over 150 billion pieces of mail in 2013 far too much for efficient human sorting. But as recently as 1997, only 10% of hand-addressed mail was successfully sorted automatically.

17 The challenge in automation is enabling computers to interpret endless variation in handwriting.

18 By providing feedback, the Postal Service was able to train computers to accurately read human handwriting. Today, with the help of machine learning, over 98% of all mail is successfully processed by machines.

19 AZURE MACHINE LEARNING CASE : THYSSEN KRUPP

SQL Server New innovations. Ivan Kosyakov. Technical Architect, Ph.D., Microsoft Technology Center, New York

SQL Server New innovations. Ivan Kosyakov. Technical Architect, Ph.D., Microsoft Technology Center, New York 2016 New innovations Ivan Kosyakov Technical Architect, Ph.D., http://biz-excellence.com Microsoft Technology Center, New York The explosion of data sources... 25B 1.3B 4.0B There s an opportunity to drive