IBM Balanced Warehouse Solution for DB2 Datawarehouses. May IBM Information Management. Jacques Milman.

Transcription

1 IBM Balanced Warehouse Solution for DB2 Datawarehouses May 2007 Jacques Milman

2 Agenda IBM Dynamic Warehouse DB2 Warehouse 9 IBM Balanced Warehouse References 1

3 IBM Information Management Why is it a challenge for organizations to leverage information effectively? Information distributed in silos across the organization Volume and variety of information increasing Velocity of business driving real-time requirements Not accurate Not complete Not trusted Not timely Increased need to aggregate and analyze information dynamically 2

4 Creates challenges for traditional warehousing Not just for traditional query and reporting purposes anymore Warehouses must now: Address expanding needs for analytics and information on demand Leverage ALL types of information, including unstructured Serve increasing numbers and types of applications and users, with varying service level demands Increasingly mixed workload environments and the constantly changing needs of different business constituents require more dynamic warehousing capabilities 3

5 How IBM Enables Dynamic Warehousing Integrated offerings to enable information on demand Search & Text Analytics OmniFind Analytics Edition Enterprise Data Modeling Rational Data Architect Information Integration Information Server IBM DB2 Warehouse MDM WS Customer Center WS Product Center Process Mgmt FileNet BPM WebSphere BPM Industry Perspective IBM industry data models SOA Infrastructure 4

6 Offerings to Enable Dynamic Warehousing What are we selling DB2 Warehouse Optimized to address dynamic warehousing requirements Embedded analytics capabilities IBM Balanced Warehouse TM Appliance offerings for the entire spectrum of the market, SMB to Enterprise Industry Solutions Industry data models Complete Information Platform Extended infrastructure for information integration, master data management, enterprise data modeling, search & text analytics, process management tightly integrated Extended Warehousing Capabilities for System z Query & reporting feature enhancement and performance improvements Services Offerings GBS strategic planning & design, along with industry expertise for dynamic warehousing GTS implementation assistance 5

7 IBM DB2 Warehouse software A complete, integrated platform Embedded analytics Modeling and design Data mining and visualization Data partitioning Performance optimization Workload control In-line analytics Deep compression Data movement and transformation Administration and control Database management IBM DB2 Warehouse 6

8 Business Intelligence Solutions DB2 9 : highly scalable from Uniprocessor to MPP Support for VLDB Scalability Parallel Everything (MPP support) Integrated DWH/BI-Functions Open Enterprise Solution Full 64 bits support Deep Compression Cluster Samefunctions All architectures - all platforms Massively Parallel Processing SQL Parallel SQL SMP Mono-processor SQL SQL SQL SQL Parallel transactions Platforms : AIX HP/UX Solaris Linux Windows Copyright IBM Corporation 2006

9 DB2 shared nothing architecture select from table Tables FCM network Engine Engine Engine Engine data+log data+log data+log data+log Partition 1 Partition 2 Partition 3 Partition n Database Partitioned Database Model Database is divided into multiple partitions Partitions can run on different nodes Each Partition Server has dedicated resources Parallel Processing occurs on all partitions and is coordinated by the DBMS Single system image to user and application 8

10 Business Intelligence Solutions DB2 database parallelism & topology OS high-speed network OS OS OS MEM MEM MEM MEM Single One O/S instance/node Small nodes One DB2 partition/node SMP Server One O/S : several S Several DB2 partitions/node OS table OS Cluster of SMP High-speed network OS MEM MEM MEM table table Copyright IBM Corporation 2006

11 IBM Software Group Business Intelligence Solutions Parallel query processing 46 select connect sum(x) from table_a,table_b where a = b Sum Join Read A Read B Optimize Coord sum( ) Get statistics Catalog Access rights Statistics sum=10 sum=12 sum=13 sum=11 Agent Agent Agent Agent Part1 A Sum Join B Part2 A Sum Join B Part3 A Sum Join B PartN Sum Join A B table_a table_b Copyright IBM Corporation 2006

12 IBM Software Group Business Intelligence Solutions Mapping physical MPP architecture to DB2 shared-nothing architecture Users network DB2 Coordinator Partition Private network (FCM) DB2 Partition DB2 Partition DB2 Partition DB2 Partition DB2 Partition DB2 Partition DB2 Partition DB2 Partition I/O Channels I/O Channels Copyright IBM Corporation 2006

13 IBM Software Group Business Intelligence Solutions All tables are partitionned Big table 1 Big table 2 Small table 1 Partition 1 Partition 2 Partition 3 DB2 Database Very small tables usually not partitionned and stored in a single DB2 partition Copyright IBM Corporation 2006

14 IBM Software Group Business Intelligence Solutions Automatic data distribution using hash partitionning DISTRIBUTE BY Insert/Load HASH (trans_id) SALES Trans_id Region Type Date_Sales ColX ColY 1001 N H S L S L N H CREATE TABLE sales(col1, col2, col3, ) DISTRIBUTE BY (trans_id) Table Partition 1 Partition 2 Partition 3 DB2 Database Copyright IBM Corporation 2006

15 Business Intelligence Solutions Range partitionning DISTRIBUTE BY Insert/Load HASH (trans_id) CREATE TABLE sales(col1, col2, col3, ) DISTRIBUTE BY (trans_id) PARTITION BY RANGE(date_sales) (STARTING 1/1/2004, ENDING MAXVALUE EVERY 1 YEAR) Table Partition 1 Partition 2 Partition PARTITION BY RANGES (date_sales) DB2 Database Copyright IBM Corporation 2006

16 Business Intelligence Solutions Managing rolling windows Table Table Table ATTACH DETACH ALTER TABLE sales ATTACH PARTITION part_2007 FROM TABLE New_Sales ALTER TABLE sales DETACH PARTITION part_2004 INTO table_2004 Partition 1 Partition 2 Partition 3 DB2 Database Copyright IBM Corporation 2006

17 Business Intelligence Solutions MDC : physically clustering along logical dimensions CREATE TABLE sales(col1, col2, col3, ) DISTRIBUTE BY (trans_id) ORGANIZE BY (region,type); N,H N,L DISTRIBUTE BY HASH S,H S,L Table N,H N,L N,H N,L N,H N,L S,H S,L S,H S,L S,H S,L Partition 1 Partition 2 Partition n ORGANIZE BY MDC Copyright IBM Corporation 2006

18 Business Intelligence Solutions MDC - Row index vs Block index Index Row index 1 index entry per row Large Block index Very compact, fit in memory Only one entry per block No reorg Faster inserts/deletes Index(region) N S Copyright IBM Corporation 2006

19 Business Intelligence Solutions All together : Hash + MDC + Range partitioning DISTRIBUTE BY HASH CREATE TABLE sales(col1, col2, col3, ) DISTRIBUTE BY (trans_id) PARTITION BY RANGE(date_sales) (STARTING MINVALUE ENDING 12/31/2004, STARTING 1/1/2005 ENDING 12/31/2005, STARTING 1/1/2006 ENDING MAXVALUE) ORGANIZE BY (region,type); Table N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L N,H N,L S,H S,L PARTITION BY RANGES ORGANIZE BY Partition 1 Partition 2 Partition 3 MDC Copyright IBM Corporation 2006

20 Business Intelligence Solutions Deep compression (Ziv & Lempel) No Compression (vmstat output) Row Compression (vmstat output) 100% 100% cpu % 80% 60% 40% 20% idle iow ait sys usr cpu % 80% 60% 40% 20% idle iow ait sys usr 0% 0% seconds seconds Tablescan No Compression Row Compression # of Rows # of Pages MB Run time (secs) 90,993 43,268 MB/sec 87,847 76,972 Total user cpu time (sec,ms) 48,91 94,49 Time waited for prefectch user cpu time/row 0, , user cpu time/page 0, , rows/page 59 1 % Pages Saved: 76.4% Copyright IBM Corporation 2006

21 IBM Software Group Business Intelligence Solutions I/O parallelism select from table Table (logical) FCM network Partition 1 Partition 2 Partition 3 Partition 4 I/O I/O I/O I/O I/O I/O I/O I/O Server(s) hdisk1 hdisk2 hdisk1 hdisk2 hdisk1 hdisk2 hdisk1 hdisk2 Array 1 Array 2 Array 1 Array 2 Array 1 Array 2 Array 1 Array 2 Storage DB2 Database Copyright IBM Corporation 2006

22 Business Intelligence Solutions Simplicity & Best Practices the IBM Balanced Warehouse Balanced Configuration Unit (BCU) A minimum replicable HW/SW stack necessary to start or expand the infrastructure of a Business Intelligence (BI) system. pseries Server cpu 1 cpu n-1 cpu 2 cpu n I/O Channels pseries Server cpu 1 cpu n-1 cpu 2 cpu n I/O Channels pseries Server cpu 1 cpu n-1 cpu 2 cpu n I/O Channels BCU 1 BCU 2 BCU N The BCU captures and packages IBM s BI Best Practices in a smart & simple design philosophy and solution package Balanced: Balanced performance from each component (database, server and storage) ensures the total solution achieves optimum performance Configuration: Based on best practices and extensive certification, IBM delivers a solution that is pre-configured and ready to go without the need for client guesswork Unit: Design based on scalable building blocks allowing the data warehouse to grow in incremental, predictable and measured units over the lifecycle of the solution Why Did We Build It? Business requires fast time to value from new solutions and the BCU decreases selection, installation and integration costs of the data warehouse components Copyright IBM Corporation 2006

23 What does developing a BCU entail? Choosing the hardware stack Based on previous experience from benchmarks (such as TPC-H) Analysing specs, features, functions of the hardware Customer preferences, sales teams (pseries, storage, software) preferences Choosing the software stack DWE Base Edition (with DB2 DPF) Other db2 features are optional (e.g., SOF, POF) OS levels (AIX, Linux RHEL/SLES) Design Validation Understand / validate the performance, available, reliability characteristics of the components Such as disk performance, filesystem performance, tablescan behaviour, NIC bandwidth, etc. Throughput validation, HA configuration setup and scripts, historical comparison with prev BCUs Performance Validation Single-stream tests, multi-stream tests, utilities tests Provides data for sizing. Follow-on questions from customers / BI architects about performance characteristics of the solution System Test Validation Stress testing the solution, validated with the Active Datawarehouse a real customer workload Includes multi-user tests, ETL workload, abnormal termination tests, backup / restore, etc. Solutions Integration Validation Uses Basel II solution, tests the rest of the BI stack (DWE features). Documentation A complete document detailing the hardware and software configuration Physical layout for hardware, DB2, OS / DB2 parameters, naming conventions, etc. Best practices on administration, backup / restore, etc. 22

24 IBM Balanced Warehouse What is a BCU? A BCU (..or Balanced Configuration Unit) is a pre-configured, pre-tested allocation of server, storage and software resources, balanced for optimum performance. Data BCU: Is the modular scalable unit that grows the IBM Balanced Warehouse. Admin BCU: Is the static unit that manages the Balanced Warehoused and Data BCUs System x x Dual Core AMD System p 575+ (POWER5+) 8 Processor System Storage DS4800 System Storage DS4800 E6000 Data BCU DB2 Warehouse (4 Licenses) E7000 Data BCU DB2 Warehouse (8 Licenses) 1 TB of Raw Data 2 TB of Raw Data Note: Balanced Warehouses will scale in sizing units associated with the Data BCU. 2 TB of Raw Data With our current BCUs this would be 1 and 2 TB units, for Linux and AIX respectively. 23

25 IBM Balanced Warehouse What is a Balanced Warehouse? AIX Example The IBM Balanced Warehouse is the complete IBM data warehousing solution comprised of multiple BCUs (Balanced Configuration Units) that are pre-configured and balanced for optimum performance. 1 Data BCU Server + Storage Balanced Warehouse (1) AIX Admin BCU 2 1 C-Class currently do not scale through the addition of BCUs due to their specialized form factor, targeting smaller installations. 2 Admin BCUs are used to provide catalog and coordination functionality to the warehouse are not used to grow the Balanced Warehouse (3) AIX Data BCUs Note: Growth occurs through the simple addition of Data BCUs to the Balanced Warehouse. 2 TB 2 TB 2 TB 6TB IBM Balanced Warehouse expand 2 TB With this addition, the Balanced Warehouse would now have 8TB of usable storage 24

26 IBM Balanced Warehouse Offering Roadmap 1H 2007 Application Solutions Growth Solutions Enterprise Solutions Solution Name C1000 C3000 D5000 E6000 E7000 E7050 Operating System Novell Novell Novell Novell / Red Hat Scalable Units N/A N/A BCU BCU BCU BCU Scaling Increments ~ 50GB ~ 250GB 1TB 1TB 2TB 2TB DB2 Warehouse Starter Intermediate Adv, Base or Ent Base or Enterprise System x3500 (I) x3500 (I) x3650 (I) x3455 (A) p5-575 p5-575 Storage Indiv or EXP EXP3000 EXP3000 DS4800 DS4800 DS8100 Support Opt. Support via Channel BCU Support BCU Support Sales Channel Channels, VAR/VAD, SI IBM Direct Sales Only Performance 25

27 Balanced Warehouse Scalability Industry Application Scaling Description Scale Result Insurance Financial Distribution Telco 2 BCUs to 4 BCUs; 4 workloads 4 BCUs to 5 BCUs; 1 workload 2 BCUs to 3 BCUs; 3 workloads 6 BCUs to 12 BCUs; 5 workloads 95% to 109% 107% 85% to 112% 105% to 122% 26

28 Linux BCU performance numbers Good scalability on multi-user, complex query tests Average I/O throughput for multiuser runs over 85% of max for 5 stream 65% of max during 20 stream Average IOPS at 8000 during 20 stream run Good disk utilization with 20 streams average 7% iowait throughout run 27

29 IBM Balanced Warehouse A ready-to-go warehouse that is simple, flexible and optimized Simple: Flexible Reduced Complexity: Ships configured and Ready to go Pre-tested: Validated by IBM Full integration: High Performance DB2 Warehouse delivered load ready Predictable, modular growth through the usage of BCU 1 One number: Single point of contact for all support issues 2 Modular scalability: Grows with your demands, not your vendor s Open and Reusable: Ensures that you are not locked in to proprietary limitations Spectrum of offerings: Multiple on-ramps to warehousing Ability to plan for and price the growth of your company s warehousing needs Optimized: IBM Balanced Warehouse solutions are thoroughly tested and tuned for performance Solutions developed through IBM Best Practices of successful client implementations Low Risk: Guaranteed performance 1 C-Class offerings do not scale through the addition of BCUs due to their specialized form factor, targeting smaller installations. 2 C-Class offerings are supported through associated reseller/distributor service agreements 28

30 Some of IBM s DWH/BCU Customers 100TB+ 50TB 20TB 29

31 IBM Software Group Business Intelligence Solutions Sprint BCU Implementation Today- Production The V4 implementation uses the AIX BCU. The current configuration is: 14 p5-575 Data BCUs (nodes) for the large DB2 tables 8 processors per node 32G memory per node 8 DB2 DPF partitions (BPUs) per data node = 112 total partitions for large DB2 tables 20 physical disks per partition for data, indexes, and logs DB2 was initially installed and implemented with 8.1.8a + one special build DB2 Partitions AIX OS image 8 s 32GB of memory Communication I/O Channels 1 p5-575 Admin BCU (1 partition for catalog, QP, small tables, + 2 additional partitions to allow for parallel load splitting) 1 p5-575 ETL BCU (4 partitions to allow for parallel load splitting and one of those also serves as coordinator for Insert and Delete activity) 2G CDR Deletes 80% faster 3G Data record Deletes 80% faster Multi Dimensional Clustering (MDCs) made this possible Storage Capacity AIX BCU Copyright IBM Corporation 2006

32 Chase Manhattan Bank!"# $# %%" &#'"# $# " () *)+!"&, - ). /- #) '0) 1 %" # 31

33 Linkshare : 10TB Linux Balanced Warehouse 16 IBM e325servers, 16 s 4 IBM DS4500 Storage Servers 224 Drives, each 73GB 15Krpm 2Gbps FC HS 14 Terabytes usable storage after Raid5 Ethernet Switch DB2 V8.2 Linux OS SLES 8 SP3 Node Configuration: 325 2Gbps Fibre Channel 1Gbps Ethernet (private) 1Gbps Ethernet (public) Processors 1 x 2.0GHz AMD 64-bit Opteron Model 248 Memory 4 x 1GB PC2700 CL2.5 ECC DDR SDRAM RDIMM Disk Controllers 2 x QLA2312 HBA (2 ports x 2G bps FC) Disk Drives 1 x 72GB 15Krpm Ultra320 SCSI HS HDD 0.25 x IBM DS4500 Storage Server 1 x EXP700 ( x 14 73GB 15Krpm 2Gbps FC HS HD) 32

34 33