The World of BI is Changing

Transcription

1 Module 5: Best Practices for Scaling With Enterprise Needs Copyright R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit written permission of the copyright owners. for by Rick F. van der Lans R20/Consultancy BV The World of BI is Changing New data sources New user groups New forms of reporting and analytics Operational BI More agility required Time-to-market TCO has to go down Copyright R20/Consultancy B.V., The Hague, The Netherlands 2 1

2 Network of Databases (Labyrinth) personal data stores databases data marts data staging area operational data store data warehouse Copyright R20/Consultancy B.V., The Hague, The Netherlands 3 Data Virtualization application reporting & analytics SOA SQL statement SQL statement Data Virtualization SQL statement Server Copyright R20/Consultancy B.V., The Hague, The Netherlands 4 2

3 Summary of Modules 1-4 application reporting & analytics Data Virtualization Server SOA Data virtualization leads to simplification of BI systems which increases agility and productivity, and improves maintenance Data modeling becomes more collaborative and iterative On-demand data profiling leads to more collaborative development What self-service BI tools do for the reporting layer (increase agility), that s what data virtualization tools do for the data storage layer Copyright R20/Consultancy B.V., The Hague, The Netherlands 5 Enterprise Needs Advanced caching mechanism High query performance Wide range of query optimization techniques Extensive data security features Alternative data integration forms Seamless switching between data integration techniques Copyright R20/Consultancy B.V., The Hague, The Netherlands 6 3

4 Speeding Up a Data Virtualization Server Data virtualization server Analytical in-memory database server databases data staging area other data sources data store data warehouse Copyright R20/Consultancy B.V., The Hague, The Netherlands 7 Caches Mimimize Access to Data Stores Virtual table without cache Virtual table with cache Copyright R20/Consultancy B.V., The Hague, The Netherlands 8 4

5 When to Use Caching? Query performance Load optimization Consistent reporting Source availability Complex transformations Security Copyright R20/Consultancy B.V., The Hague, The Netherlands 9 Caches on Multiple Levels V 2 V 3 V 1 Copyright R20/Consultancy B.V., The Hague, The Netherlands 10 5

6 Different Forms of Caching Full refreshing, incremental refreshing, live refreshing Online and Offline refreshing Time-based caching, eventbased caching, and query-based caching Cache replication Copyright R20/Consultancy B.V., The Hague, The Netherlands 11 Refresher of Query Optimization (1) Processing Strategy 1 SELECT * FROM CUSTOMER WHERE CUSTOMER_ID = 45 RESULT := []; FOR EACH C IN CUSTOMER DO IF C.CUSTOMER_ID = 45 THEN RESULT :+ C; ENDFOR; Processing Strategy 2 RESULT := []; FOR EACH C IN CUSTOMER WHERE CUSTOMER_ID = 45 DO RESULT :+ C; ENDFOR; Copyright R20/Consultancy B.V., The Hague, The Netherlands 12 6

7 Six Stages of Query Optimization Application Query acquisition 1 6 Result Transmission 2 Query optimization by DBS 3 Data access Database Server Database 5 Database processing 4 Data retrieval Copyright R20/Consultancy B.V., The Hague, The Netherlands 13 Refresher of Query Optimization (1) Processing Strategy 1 SELECT * FROM CUSTOMER WHERE CUSTOMER_ID = 45 RESULT := []; FOR EACH C IN CUSTOMER DO IF C.CUSTOMER_ID = 45 THEN RESULT :+ C; ENDFOR; Processing Strategy 2 RESULT := []; FOR EACH C IN CUSTOMER WHERE CUSTOMER_ID = 45 DO RESULT :+ C; ENDFOR; Copyright R20/Consultancy B.V., The Hague, The Netherlands 14 7

8 Refresher of Query Optimization (2) Processing strategy 1 SELECT CUSTOMER_ID, CITY_NAME FROM CUSTOMER WHERE CUSTOMER_ID < 10 AND CITY_NAME = 'Tulsa' ORDER BY CUSTOMER_ID RESULT := []; FOR EACH C IN CUSTOMER DO IF (C.CUSTOMER_ID < 10) AND (C.CITY_NAME = 'Tulsa') THEN RESULT :+ C; ENDFOR; RESULT := []; Processing strategy 2 FOR EACH C IN CUSTOMER WHERE CUSTOMER_ID < 10 DO IF C.CITY_NAME = 'Tulsa' THEN RESULT :+ C; ENDFOR; RESULT := []; FOR EACH C IN CUSTOMER WHERE CITY_NAME = 'Tulsa' DO IF C.CUSTOMER_ID < 10 THEN RESULT :+ C; ENDFOR; Copyright R20/Consultancy B.V., The Hague, The Netherlands 15 Ten Stages of DV Query Optimization Application Query acquisition 1 10 Result transmission Query optimization by DVS Query shipment Query optimization by DBS Data access Data Virtualization Server Database Server Database Data transformation and federation Data Transmission Database processing Data retrieval Copyright R20/Consultancy B.V., The Hague, The Netherlands 16 8

9 Query Substitution SELECT * FROM V1 V 1 V1 mapping: SELECT C1, COUNT(*) FROM V2 GROUP BY C1 V 2 V2 mapping: SELECT C1, C2, C3 FROM T1 WHERE C2 > 1000 T 1 Executed query: SELECT C1, COUNT(*) FROM T1 WHERE C2 > 1000 GROUP BY C1 Copyright R20/Consultancy B.V., The Hague, The Netherlands 17 Query Pushdown (1) Query: SELECT C2, CONCAT(C5, C6) FROM T1 WHERE C1 = > 1000 AND C4 BETWEEN 10 AND 20 V 1 V1 Mapping: SELECT * FROM V1 T 1 Pushed down query: SELECT C2, C5, C6 FROM T1 WHERE C1 = > 1000 AND C4 BETWEEN 10 AND 20 Copyright R20/Consultancy B.V., The Hague, The Netherlands 18 9

10 Big table Small table Query Pushdown (2) Query: SELECT T1.C1, T2.C1, T3.C4 FROM T1, T2, T3 WHERE T1.C1 = T2.C2 AND T2.C1 = T3.C3 AND T2.C1 > 1000 AND T3.C4 BETWEEN 10 AND 20 Query to database 1: SELECT T1.C1, T2.C1 FROM T1, T2 WHERE T1.C1 = T2.C2 AND T2.C1 > 1000 V 1 V 2 V 3 V 3 Query to database 2: SELECT T3.C3, T3.C4 FROM T3 WHERE T3.C3 > 1000 AND T3.C4 BETWEEN 10 AND 20 T 1 T 2 T 3 Copyright R20/Consultancy B.V., The Hague, The Netherlands 19 Query Expansion (Injection) Query: SELECT FROM WHERE T1.C1, T1.C2 T1, T2 T1.C1 = T2.C2 Expanded query: SELECT T1.C1, T1.C2 FROM T1 WHERE T1.C1 IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20) Copyright R20/Consultancy B.V., The Hague, The Netherlands 20 10

11 Different Types of Data Integration ETL/ELT Data replication On-demand data integration On-demand data integration with cache Copyright R20/Consultancy B.V., The Hague, The Netherlands 21 Seamless Switching Between Data Integration Forms Data Virtalization Repository With Integration Specifications ETL/ELT Data replication On-demand data integration On-demand data integration with cache Copyright R20/Consultancy B.V., The Hague, The Netherlands 22 11

12 Data Virtualization & Data Replication application operational reporting replication database replicated database Copyright R20/Consultancy B.V., The Hague, The Netherlands 23 Data Virtualization & Data Replication operational reporting application data virtualization server replication database replicated database Copyright R20/Consultancy B.V., The Hague, The Netherlands 24 12

13 Data Protection Copyright R20/Consultancy B.V., The Hague, The Netherlands 25 Row-based Data Protection Copyright R20/Consultancy B.V., The Hague, The Netherlands 26 13

14 Summary Advanced data virtualization means high performance, advanced data security, and support for multiple data integration styles Analysis & Design Development Runtime & Management Data federation Data virtualization Advanced data virtualization Copyright R20/Consultancy B.V., The Hague, The Netherlands 27 Five Modules Module 1: How to Architect for Agility and Productivity in Your Projects Module 2: Best Practices for Developing a Reusable and Flexible Data Model Module 3: Best Practices for Real-Time Data Profiling and Data Quality Module 4: Best Practices for Ensuring Managed Self-Service BI Module 5: Best Practices for Scaling with Enterprise Needs Copyright R20/Consultancy B.V., The Hague, The Netherlands 28 14

15 Rick F. van der Lans Rick F. van der Lans is an independent consultant, lecturer, and author. He specializes in data warehousing, business intelligence, service oriented architectures, and database technology. He is managing director of R20/Consultancy B.V.. Rick has been involved in various projects in which SOA, data warehousing, and integration technology was applied. Rick van der Lans is an internationally acclaimed lecturer. He has lectured professionally for the last twenty years in many of the European and Middle East countries, the USA, South America, and in Australia. He has been invited by several major software vendors to present keynote speeches. He is the author of several books on computing, including his new Data Virtualization for Business Intelligence Systems. Some of these books are available in different languages. Books such as the popular Introduction to SQL is available in English, Dutch, Italian, Chinese, and German and is sold world wide. He also authored The SQL Guide to Ingres and SQL for MySQL Developers. As author for BeyeNetwork.com, writer of whitepapers, as chairman for the annual European Data Warehouse and Business Intelligence Conference, and as columnist for a few IT magazines, he has close contacts with many vendors. R20/Consultancy B.V. is located in The Hague, The Netherlands, You can get in touch with Rick via: rick@r20.nl LinkedIn: Copyright R20/Consultancy B.V., The Hague, The Netherlands 29 15