Agile Data Integration for Business Intelligence Lecture Series

Transcription

1 Agile Data Integration for Business Intelligence Lecture Series Copyright R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit written permission of the copyright owners. for by Rick F. van der Lans R20/Consultancy BV Module 3: Best Practices for Real-Time Data Profiling and Data Quality Copyright R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit written permission of the copyright owners. for by Rick F. van der Lans R20/Consultancy BV 1

2 The World of BI is Changing New sources New user groups New forms of reporting and analytics Operational BI More agility required Time-to-market TCO has to go down Copyright R20/Consultancy B.V., The Hague, The Netherlands 3 Network of Databases (Labyrinth) personal stores production bases marts staging area operational store warehouse Copyright R20/Consultancy B.V., The Hague, The Netherlands 4 2

3 Data Virtualization production application reporting & analytics SOA SQL statement SQL statement Data Virtualization SQL statement Server Copyright R20/Consultancy B.V., The Hague, The Netherlands 5 Summary of Modules 1 and 2 production application reporting & analytics Data Virtualization Server SOA Data virtualization leads to simplification of BI systems which increases agility of BI systems and productivity, and improves maintenance Data modeling becomes more collaborative and iterative Centralized implementation of common integration specifications leads to a higher level of reuse Copyright R20/Consultancy B.V., The Hague, The Netherlands 6 3

4 What is Data Profiling? Wikipedia: Data profiling is the process of examining the available in an existing source (e.g. a base or a file) and collecting statistics and information about that. Classic profiling: Source Extracted Profiling results Analyst Copyright R20/Consultancy B.V., The Hague, The Netherlands 7 The Virtual Table (LDO) Virtual table with common specifications Source table Data integration specifications Homogeneous and heterogeneous Transformations Standardization of structure and values Cleansing operations From simple to complex Aggregations Copyright R20/Consultancy B.V., The Hague, The Netherlands 8 4

5 On-Demand Viewing of Virtual Content Copyright R20/Consultancy B.V., The Hague, The Netherlands 9 On-Demand Data Profiling Copyright R20/Consultancy B.V., The Hague, The Netherlands 10 5

6 On-Demand Data Profiling Copyright R20/Consultancy B.V., The Hague, The Netherlands 11 Special Interface for Business Users Copyright R20/Consultancy B.V., The Hague, The Netherlands 12 6

7 Advantages of On-Demand Data Profiling On-demand profiling Iterative profiling Collaborative profiling No loss of time due to switching between tools Copyright R20/Consultancy B.V., The Hague, The Netherlands 13 Enforcing Data Quality with Data Virtualization Different types of incorrect Missing Misspelled False Different options Nothing Filtering of rows Filtering of values Flagging Restoring Copyright R20/Consultancy B.V., The Hague, The Netherlands 14 7

8 Technique 1: Filtering Incorrect Data Correct Correct and incorrect Copyright R20/Consultancy B.V., The Hague, The Netherlands 15 Technique 2: Flagging Incorrect Data Correct and flagged incorrect Correct and incorrect Copyright R20/Consultancy B.V., The Hague, The Netherlands 16 8

9 Technique 3: Restoring Incorrect Data Correct and restored Correct and incorrect Copyright R20/Consultancy B.V., The Hague, The Netherlands 17 Upstream Data Cleansing production bases staging area warehouse marts personal stores Copyright R20/Consultancy B.V., The Hague, The Netherlands 18 9

10 Impact of Data Virtualization on Data Quality Less Proliferation of Meta Data Specifications Less Copies of the Data Centralized Data Access Improved Data Consistency Improved Auditability Improved Flexibility Copyright R20/Consultancy B.V., The Hague, The Netherlands 19 Evolution: From Data Federation to Advanced Data Virtualization Analysis & Design Development Runtime & Management Data federation Data virtualization Advanced virtualization Copyright R20/Consultancy B.V., The Hague, The Netherlands 20 10

11 Advanced Data Virtualization is More developer-friendly development-environment Support for self-service, iterative, and collaborative development More support for (canonical) modeling, On-demand profiling, and cleansing More enterprise-ready: high-performance, scalability, security Architecturally strategic Full support for entire development life cycle: business glossary, information modeling Extensive integrity features Extensive master management features Full integration of different integration styles, including ETL, ELT, and replication Copyright R20/Consultancy B.V., The Hague, The Netherlands 21 Summary Data virtualization makes profiling more collaborative and interactive activities Improves correctness of and correctness of reporting results Data quality rules can be enforced with virtualization Copyright R20/Consultancy B.V., The Hague, The Netherlands 22 11

12 Five Modules Module 1: How to Architect for Agility and Productivity in Your Projects Module 2: Best Practices for Developing a Reusable and Flexible Data Model Module 3: Best Practices for Real-Time Data Profiling and Data Quality Module 4: Best Practices for Ensuring Managed Self-Service BI Module 5: Best Practices for Scaling with Enterprise Needs Copyright R20/Consultancy B.V., The Hague, The Netherlands 23 Rick F. van der Lans Rick F. van der Lans is an independent consultant, lecturer, and author. He specializes in warehousing, business intelligence, service oriented architectures, and base technology. He is managing director of R20/Consultancy B.V.. Rick has been involved in various projects in which SOA, warehousing, and integration technology was applied. Rick van der Lans is an internationally acclaimed lecturer. He has lectured professionally for the last twenty years in many of the European and Middle East countries, the USA, South America, and in Australia. He has been invited by several major software vendors to present keynote speeches. He is the author of several books on computing, including his new Data Virtualization for Business Intelligence Systems. Some of these books are available in different languages. Books such as the popular Introduction to SQL is available in English, Dutch, Italian, Chinese, and German and is sold world wide. He also authored The SQL Guide to Ingres and SQL for MySQL Developers. As author for BeyeNetwork.com, writer of whitepapers, as chairman for the annual European Data Warehouse and Business Intelligence Conference, and as columnist for a few IT magazines, he has close contacts with many vendors. R20/Consultancy B.V. is located in The Hague, The Netherlands, You can get in touch with Rick via: rick@r20.nl LinkedIn: Copyright R20/Consultancy B.V., The Hague, The Netherlands 24 12