Data Warehousing on System z Best Practices

Transcription

1 Data Warehousing on System z Best Practices Willie Favero, IBM DB2 SME Data Warehousing for System z Swat Team IBM Silicon Valley Laboratory Wednesday, May 16, 2013

2 Please Note: IBM s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. Slide 1 of 80

3 Acknowledgements and Disclaimers: Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Copyright IBM Corporation U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others. Slide 2 of 80

4 What Do I Mean by Best Practices This presentation consist of Stuff I have been asked about o Stuff I ve been told about that made sense Things I have tried that have worked o Some things I ve tried that haven t worked Techniques I ve read about that I thought should be here Please consider this presentation just a starting point, not the conclusion All put here, in one place, for you to decide what s good (can stay) and what isn t (has to go) Slide 3 of 80

5 Here s My Top 10 Best Practices Partitioning Parallelism Data compression Index Compression (maybe not) Star (Snowflake) Schema Query Processing Manage statistics Maintain accounting and statistical data Are backups necessary DSNZPARMs DFSORT Buffer Pools And yes, I do realize there are more than 12 (smile) Slide 4 of 80

6 Here s My Top 10 Best Practices Partitioning 1 Parallelism Data compression Index Compression (maybe not) Star (Snowflake) Schema Query Processing Manage statistics Retain/manage accounting and statistical data Are backups necessary DSNZPARMs DFSORT Buffer Pools Slide 5 of 80

7 Partitioning For large objects (????) Manageability Potential performance improvements o Parallelism # CPs and # of Partitions o Scan only necessary partitions Should almost always be considered for fact table o Use date key for fact table partitioning if available You could have up to 4096 partitions Slide 6 of 80

8 Partitioning Key Date column if available Year? Quarter? Month? Day even? Sometimes daily partitions can be attractive option o Could be easier to manage Surrogate key if not a usable date column Use integer; short key Use BIGINT; if more values are necessary Slide 7 of 80

9 Partition Pruning SELECT FROM part_table WHERE LASTNAME = favero Partition by date Part 1 Part 2 Part 3 Part 4 Part x Part x+1 Part x+n favero favero favero Slide 8 of 80

10 What About Universal Table Spaces? Introduced in DB2 9 Improved in DB2 10 Two flavors Partition-by-growth o No partitioning key definition Range-partition o Like legacy partitioning but uses segmentation The best of segmented and partitioning in one object Both flavors can have to 128 TB of data (as of V8) Both have mass DELETE Better space management (improved space maps) Slide 9 of 80

11 Table Space Definition No or low INSERT/UPDATE rate FREEPAGE 0 PCTFREE 0 APPEND YES clause on CREATE/ALTER TABLE SECQTY set to -1 PRIQTY consider setting to -1 LOCKSIZE CLOSE YES or CLOSE NO NOT LOGGED Isn t a warehouse usually READ ONLY Slide 10 of 80

12 Table Space Definition SEGSIZE If > 0, range-partitioned universal table space if NUMPARTS is also specified 0 and NUMPARTS specified, legacy partitioned table space DSSIZE Defines maximum size of each partition 1G, 2G, 4G, 8G, 16G, 32G, 64G, 128G, 256G NUMPARTS Indicates partitioned table space 1 to 4086 partitions can be specified Page size and DSSIZE drive max NUMPARTS value Slide 11 of 80

13 What About Dimension Tables? They have design considerations also Use integer surrogate keys if date column not available Using integer prevents fact table from unnecessary width May have to plan for INSERT, UPDATE, and DELELTE activity (Fact table tends to be loaded) Slide 12 of 80

14 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data compression Index Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage statistics Retain/manage accounting and statistical data Are backups necessary DSNZPARMs DFSORT Buffer Pools Slide 13 of 80

15 CP Parallelism (Behind the Scenes) Multi-Tasking - How does DB2 do it? Spawning parallel tasks: z/os preemptable SRBs are used for work done in parallel. Originating Task (TCB) handles SRB creation, cleanup and data merging. Originating Task TCB SRB SRB SRB Parallel Tasks Preemptable SRBs: o Synchronize originating and parallel tasks o Introduced with Enclave Services (MVS 5.2) o Inherit dispatching priority of allied address space. Therefore all work is done at the same priority (goodness) Originating task does not control scheduling or which CP an SRB is run on z/os handles scheduling. DB2 handles synchronization through suspending and resuming tasks Slide 14 of 80

16 CP Parallelism (Behind the Scenes) Parallel tasks are started at OPEN CURSOR* Application might be able to take advantage of this to achieve inter-query parallelism: DECLARE CURSOR C1 FOR SELECT COUNT(*) FROM ORDERS WHERE INVOICE_AMT > DECLARE CURSOR C2 FOR SELECT PARTNAME FROM PARTS WHERE INVENTORY_AMT > 200 OPEN CURSOR C1 Parallel Task #1 Parallel Task #2 Parallel Task #3 OPEN CURSOR C FETCH C1 FETCH C2 Parallel Task #1 Parallel Task #2 Parallel Task #3 *Exception if RID sort, but no data sort, then //ism starts at first fetch (same as without //ism) Slide 15 of 80

17 CP Parallelism (Behind the Scenes) Parallel Degree Determination "Why is the degree sometimes less than the number of parts?" Optimal degree for parallel group is determined at BIND/PREPARE time o Also called "Planned BIND degree" - shown in EXPLAIN output Optimal degree determined by considering: o Number of table space partitions o Estimated I/O cost of largest partition o Estimated CP cost considering: Processing cost MIPS rating of machine Number of CPs on-line (used for CP parallelism only) Degree determination deferred if access path dependent on host variable Slide 16 of 80

18 Determining the degree of parallelism DB2 chooses the smallest degree that will still deliver the best possible elapsed time With the shared data model, DB2 has the flexibility to choose the degree of parallelism I/O-intensive: Degree of parallelism approaches the number of partitions Processor-intensive: Degree of parallelism approaches the number of processors (for DB2 9 - times 4, for DB2 10 times 2) CP CP CP CP CP CP CP CP CP CP X 4 (or 2) Additionally, skews in the data organization can be detected and compensated for in choosing the degree of parallelism Slide 17 of 80 Parallel Task #1 Parallel Task #3 Parallel Task Data #2 Warehousing on System z Best Practices

19 Star (Snowflake) Schema Star (snowflake) schema = a relational database schema for representing multidimensional data Sometimes graphically represented as a star or snowflake Data is stored in a central fact table Surrounding additional dimension tables hold information about each perspective of the data Example: store "facts" of the sale (units sold, price,..) with product, time, customer, and store keys in a central fact table. Store full descriptive detail for each keys in surrounding dimension tables. This allows you to avoid redundantly storing this information (such as product description) for each individual transaction Complex star schema parallel queries include the acts of joining several dimensions of a star schema data set (like promotion vs. product). Two specific DSNZPARMs must be setup accordingly: STARJOIN and SJTABLES. Proper index design must be present in the star schema tables. Slide 18 of 80

20 Star Schema Dimension Table Dimension Table Fact Table Dimension Table Dimension Table Dimension Table Star schema = Snowflake schema Slide 19 of 80

21 Star Schema Store Product Three or more tables can be joined at one time ZPARMs SJTABLES and STARJOIN must be set ziip eligible Sale Date Promotion Associate Star schema = Snowflake schema Slide 20 of 80

22 Queries Include partitioning key as WHERE predicate when possible EXPLAIN, Accounting and Statistics traces Improves limited partition scan o EXPLAIN s Page-Range column Know when to STOP tuning Take advantage of technology Table Expressions Materialized Query Tables (MQT) Common Table Expressions (CTE) Slide 21 of 80

23 Referential Integrity Good idea just be careful Apply after LOAD to improve load performance Integrity verses performance? What are the tradeoffs Maintain integrity Increase cost of INSERT, DELETE, and UPDATE processing Informational RI required for MQTs Slide 22 of 80

24 Manage Amount of Data Maintained If date partitioning key, roll off partitions Archive Consider offloading partitions no longer accessed o Use a technique that removes data from fact table but still leaves it available to queries if necessary DB2 Analytics Accelerator for z/os V3 highperformance storage saver Slide 23 of 80

25 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage statistics Retain/manage accounting and statistical data Are backups necessary DSNZPARMs DFSORT Buffer Pools Slide 24 of 80

26 DB2 Data Compression Compression should always be considered for a warehouse Savings are usually greater than 50% Have seen as high as 80% in certain situations Overhead on INSERT, minimal overhead on SELECT Warehouse queries dominated by sequential prefetch, which benefit from DB2 compression. Not all rows in a table space can be compressed If the row after compression is not shorter than the original uncompressed row, the row remains uncompressed. Compression dictionary size 64K (16 X 4K page) of storage in the DBM1 address space Dictionary goes above the 2 GB bar Faster hardware can result in faster compression Slide 25 of 80

27 Dictionary 4K Page Size Data pages (18-???) Dictionary pages (20-17) The dictionary is created by the LOAD and/or REORG utilities only It occupies: o 4K 16 pages o 8K - 8 pages o 16K - 4 pages o 32K - 2 pages Space map page (1) Header page (0) The compression dictionary follows the header and first space map pages (next slide) Dictionaries can be at the partition level (Careful, you could have 4096 partitions Slide 26 of 80

28 Data Compression Rows are compressed on INSERT For an UPDATE Expand, update, then re-compressed row UPDATE has the potential to be expensive Changes (INSERT & UPDATE) are logged in compressed format Possible reduced logging cost Active log reductions carried over to the archive logs Slide 27 of 80

29 Data Compression Larger page sizes may result in better compression. Resulting rows after compression are variable length You might be able to fit more rows with less wasted space in a larger page size. You cannot turn compression on for the catalog, directory, work files, or LOB table spaces Index compression does not use a dictionary Slide 28 of 80

30 Possible Performance Gain When compression is on, data pages are brought into buffer pool in compressed state Increasing the number of rows in the same size pool could increase buffer pool hit ratio Increasing hit ratio could reduce I/O necessary to satisfy the same number of getpage requests. If Compression doubles the number of rows per page When DB2 loads that page in a buffer pool, it will be loading twice as many rows. Less I/O is always a good thing. Slide 29 of 80

31 DB2 Index Compression.. Index compression is new to DB2 9 for z/os Page level compression Unlike data row compression: Buffers contain expanded pages Pages are decompressed when read from disk A buffer hit does not need to decompress Like data row compression: An I/O bound scan will run faster Prefetch performs the decompression asynchronously Pages are compressed by the deferred write engine DSN1COMP utility can be used to predict space savings Index compression saves space, it s not for performance Slide 30 of 80

32 Index Compression: Performance CPU cost is mostly inconsequential. Most of the cost is asynchronous, the exception being a synchronous read. The worst case is an index with a poor buffer hit ratio. Example: Suppose the index would compress 3- to-1. You have three options.. 1. Use 8K buffer pool. Save 50% of disk. No change in buffer hit ratio or real storage usage. 2. Use 16K buffer pool and increase the buffer pool size by 33%. Save 67% of disk, increase real storage usage by 33%. 3. Use 16K buffer pool, with no change in buffer pool size. Save 67% of disk, no change in real storage used, decrease in buffer hit ratio, with a corresponding increase in synchronous CPU time. Slide 31 of 80

33 ..DB2 Index Compression The CI Size of a compressed index on disk is always 4K A 4K expands into a 8K or 16K buffer, which is the DBA s choice. This choice determines the maximum compression ratio. Compression of key prefix and RID Lists A Rid List describes all of the rows for a particular index key An index with a high level of non-uniqueness, producing long Rid Lists, achieves about 1.4-to-1 compression Compression of unique keys depends on prefix commonality Slide 32 of 80

34 Slide 33 of 80

35 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Retain/manage accounting and statistical data Are backups necessary DSNZPARMs DFSORT Buffer Pools Slide 34 of 80

36 Do Not Ignore Statistics Needed only if you plan to run queries LOAD/REORG/REBUILD utilities with inline stats Careful - Do not gather all stats you may need o i.e.histogram RUNSTATS Use to gather stats for specific quer needs Use Data Studio s Statistics Advisor to determine RUNSTATS control cards o SAMPLE vs TABLESAMPLE SYSTEM (AUTO) Slide 35 of 80

37 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Retain/manage 8 accounting and statistical data Are 9 backups necessary DSNZPARMs DFSORT Buffer Pools Slide 36 of 80

38 Plan on Saving Performance Data DB2 traces Accounting records (SMF 100) o Classes 1,2 and 3 o If using packages, classes 7 and 8 Statistics records (SMF 101) o Classes 1, 3, 4, 5, and 6 Design a performance database o See Omegamon as an example EXPLAIN outputs Slide 37 of 80

39 Is a Test System Necessary Have test mimic production warehouse in as many ways as possible Statistics Cache definitions Buffer pools, sort pool, etc Profiles Number and Speed of processors DSNZPARM Slide 38 of 80

40 Are Backups Necessary? Does data need to be recoverable? Can it be reloaded with last refresh? Is warehouse data included in DR plan? Should warehouse data be included in DR plan? Slide 39 of 80

41 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Maintain 8 accounting and statistical data Are 9 backups necessary DSNZPARMs 10 DFSORT Buffer Pools Slide 40 of 80

42 DSNZPARMs Parallelism CDSSRDEF (DSN6SPRM) Update Yes Set to 1 or ANY, 1 is the default o Can affect your distributed SQL PARAMDEG (DSN6SPRM) Update Yes Controls the max degree of parallelism Set to 0 if DB2 should decide, otherwise o Pick a reasonable value based on the number of CPs PARA_EFF (DSN6SPRM) Update yes 0-100, 50 default how is parallelism effected o 1 less parallelism, 99 more parallelism SPRMPTH (DSN6SPRC) Update No Default 120, may want a higher value Slide 41 of 80

43 MXQBCE With system parameter MAX_OPT_STOR (default 40MB) DB2 will try to cap the storage usage for bind/prepare at the value set Attempting to avoid SQLCODE -904 or storage abends When DB2 gets close to the threshold it will engage in "emergency clipping Choosing the lowest cost access path at that point and continue processing it Discarding all others If the threshold is set too low, then DB2 may have only completed small% of the prepare which may not be enough to really distinguish which is the best plan Slide 42 of 80

44 MXQBCE cont Better to use system parameter MXQBCE instead Controls number of join combinations considered by the DB2 Optimiser Predictive rather than reactive Attempts to clip access paths early, so as to avoid emergency clipping Use the formula (2**n)-1 Default of (2**15)-1=32767 prepare will take no worse than a 15 table join For a 14 table join it is 6383, 13 table 8191, 12 table 4095, 11 table 2047 and 10 table 1023 Do not go below 1023 Slide 43 of 80

45 DNSZPARMs Dynamic Statement Cache CACHEDYN (DSN6SPRM) Update Yes Set to YES (default) EDMSTMTC (DSN6SPRM) Update Yes EDM Statement Cache size Installation defined based on the amount of dynamic SQL expected and how long cache will gather information before wrapping Maximum size 113,386K Rollover buffer, uses LRU algorithm Slide 44 of 80

46 DSNZPARMs Star Join STARJOIN* (DSN6SPRM) Update Yes DISABLE, ENABLE, 1, 2 to 3276 Turn on star join and how to handle fact table SJTABLES* (DSN6SPRM) Update Yes Only consider star join when number of table is great than this value * Profile tables can also be used to manage these values Slide 45 of 80

47 DSNZPARMs - Caches MXDTCACH (DSN6SPRM) Update YES Data caching per thread Default is 20, might be low, try 128 Slide 46 of 80

48 DSNZPARMs - Caches MAXTEMPS_RID (DSN6SPRM) The maximum amount of temporary storage in the work file database that a single RID list can use at a time Update YES Possible values: NONE, NOLIMIT, or 1 to Default: NOLIMIT Suggest setting to a value o Consider not using NONE or NOLIMIT If using NOLIMIT, be very cautious o You might start with something like That s about 6.5 million additional RIDs in work file Slide 47 of 80

49 DSNZPARMs - Caches SRTPOOL (DSN6SPRM) Update YES Maximum K, Default is 10000K Max out if possible but be careful, this value is per thread MAXRBLK (DSN6SPRM) Update YES Default: , Possible values are: 0, 128 to o If using MAXTEMPS_RID, do not reduce MAXRBLK Don t pick something too small because running out of space fails over to the work files Slide 48 of 80

50 DSNZPARMs - Threads CTHREAD, IDBACK, MAXDBAT, and CONDBAT, all on the DSN6SYSP macro Update Yes Manage local and distributed threads CTHREAD = Maximum number of allied (local) threads that can be concurrently allocated MAXDBAT = Maximum number of concurrent DBATs or connections if CMTSTAT=ACTIVE CONDBAT = Maximum number of concurrent connections CMTSTAT =ACTIVE or INACTIVE governs whether DBATs remain active across commits o Highly recommended to set CMTSTAT = INACTIVE Slide 49 of 80

51 DSNZPARM IDTHTOIN (DSN6FAC) Update Yes Thread timeout value Default 120 Is default long enough? 0 = never time out TCPKPALV (DSN6FAC) Update Yes TCP/IP keep alive value Default 120 Can enable, disable or set to a value Coordinate with group that maintains TCP/IP Slide 50 of 80

52 DSMAX DSMAX (DSN6SPRM) Update Yes o Default 20,000 Controls the maximum number of open datasets If 99% of DSMAX reached o Asynchronous physical VSAM close of 3% of datasets First close CLOSE=YES on LRU chain Followed by closing CLOSE=NO on LRU chain Excessive dataset open/close activity will result in high DBM1 TCB time You need to be aware of the number of data sets you are dealing with and what is affected because this uses percentages Slide 51 of 80

53 CONTSTOR Drives thread storage contraction in DBM1 when CONTSTOR = YES (DB2 v8 & DB2 9 - NO, DB YES) Associated CPU overhead (typical < 1-2%) Design point is long running persistent threads with RELEASE(COMMIT) Compresses out part of Agent Local Non-System storage Does not compress o Agent Local System o GETMAINed Stack Storage o Local Dynamic Statement Cache Controlled by two hidden system parameters (1MB) 10 (commits) Triggered at: # Commits > SPRMCTH (Agent Local Non-System > SPRMSTH & # Commits > 5) Slide 52 of 80

54 CONTSTOR Drives thread storage contraction in DBM1 when CONTSTOR = YES (DB2 v8 & DB2 9 - NO, DB YES) Associated CPU overhead (typical < 1-2%) Design point is long running persistent threads with RELEASE(COMMIT) Compresses out part of Agent Local Non-System storage Does not compress o Agent Local System o GETMAINed Stack Storage o Local Dynamic Statement Cache Controlled by two hidden system parameters (1MB) 10 (commits) Triggered at: # Commits > SPRMCTH (Agent Local Non-System > SPRMSTH & # Commits > 5) Slide 53 of 80

55 MINSTOR With MINSTOR=NO (default), first fit algorithm is used Fragmentation may happen with free space (leap frog effect) With MINSTOR=YES, best fit algorithm is used instead Will go through all the chains across all the segments Makes the storage denser Observed CPU overhead < 1% for 3-4MB storage pools Danger is that it masks storage leaks (makes them appear to go away) It makes debugging storage leaks more difficult Also degraded slower performance as the storage leak progresses Use MINSTOR=YES when System is fully tuned and optimised for storage You have determined there are no leaks Out of other options and need the last ounce of storage Slide 54 of 80

56 MGEXTSZ Enables Sliding Secondary Quantity for DB2 Managed Page sets where an explicit SECQTY value has been specified by the user and recorded in the DB2 Catalog Possible values: NO, YES Secondary extent allocations are to be optimised automatically by DB2 according to the respective sliding scale (Rule 4) DB2 will use the greater of the respective sliding scale and the secondary quantity in DB2 Catalog When allocating a new dataset for a page set DB2 uses SECQTY or DB2 calculated extent size instead of PRIQTY Slide 55 of 80

57 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Maintain 8 accounting and statistical data Are 9 backups necessary DSNZPARMs 10 DFSORT 12 Buffer 11 Pools Slide 56 of 80

58 DSNZPARMs Utility Sort Processing IGNSORTN = YES causes DB2 to ignore the SORTNUM keyword if specified. My recommendation is to leave this set to its default NO. If SORTNUM must be specified, setting to YES will prevent SORTNUM s use. Leaving IGNSORTN set to NO gives the option to specifying SORTNUM if necessary. However, it is highly recommended that you NOT use the SORTNUM keyword. Update YES Set UTSORTAL to YES, the default, allows DB2 to use real time statistics, always available in DB2 10, to determine the sort work data set sizes. NO forces you to either specify the sort work data sets in the JCL or pre-allocate the data sets. Update YES Slide 57 of 80

59 More Utility Sort Recommendations Dynamic Storage Adjustment (DSA) Initialization value that allows DFSORT to adjust the amount of main storage used by DFSORT (TMAXLIM option). The default for DSA is 64 GB, too low for a DB2 utility with large sorts. 128 GB is the suggested starting value. As high as 512 GB if terabyte sorts tables are expected. Slide 58 of 80

60 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Maintain 8 accounting and statistical data Are 9 backups necessary DSNZPARMs 10 DFSORT 12 Buffer 11 Pools Slide 59 of 80

61 Buffer Pools Catalog/directory use BP0, BP8K0, BP16K0 and BP32K0 Do NOT use for any other purpose Minimum of 4 user BPs: user index (4K) and user data (4K) and work files (4K and 32K) Don t be afraid to use 8K and 16K buffer pools In many cases can improve rows per page Separate dimension tables from fact table If dimension tables are not too large, may be able to pin in pool Same for indexes on dimension tables Slide 60 of 80

62 Buffer Pools Careful with the suggestion to Super Size your pools Always make sure you never exceed real storage DB2 10 allocates pool space as needed Be careful over allocating pool when using PGFIX = YES Consider taking advantage of 1 MB pages o Increased to 2 GB on the zec12 Requires PGFIX = YES Slide 61 of 80

63 Buffer Pools DSNDB07 buffer pools Need 4k and 32K work files o Lots of 32K, maybe more than 4k, monitor o Sort key greater than 100 bytes will usually use a 32K pool VPSEQT = 98 for DSNDB07 pools o If sparse index is used, lower to Work files Go for LARGE pools if possible o However, many and smaller is better than few and large For sort work files, always use zero secondary o However, If also using DGTT, make sure you have a few with secondary greater than 0 (zero) Be aware of how DSNZPARM keyword WFDBSEP is set (next slide) Slide 62 of 80

64 DSNZPARM WFDBSEP (DSN6SPRM) Update No Specify NO o DB2 attempts to direct declared global temporary table work to DB2-managed work file table spaces that are defined with a non-zero SECQTY. o DB2 attempts to direct sort work to DB2-managed work file table spaces that are defined with a zero SECQTY. The operative word above is Attempts o If you specify YES, Attempts becomes Always Slide 63 of 80

65 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Maintain 8 accounting and statistical data Are 9 backups necessary DSNZPARMs 10 DFSORT 12 Buffer 11 Pools Slide 64 of 80

66 DB2 Analytics Accelerator Accelerating decisions to the speed of business Blending System z and Netezza technologies to deliver unparalleled, mixed workload performance for complex analytic business needs. Get more insight from your data Fast, predictable response times for right-time analysis Accelerate analytic query response times Improve price/performance for analytic workloads Minimize the need to create data marts for performance Highly secure environment for sensitive data analysis Transparent to the application Slide 65 of 80

67 DB2 Analytics Accelerator V2 zenterprise (z196 or z114) Netezza Technology CLIENT Data Studio Foundation DB2 Analytics Accelerator Admin Plug-in OSA-Express3 Private Service Network Primary BladeCenter 10 GbE 10Gb Backup Users/ Applications Data Warehouse application DB2 for z/os enabled for IBM DB2 Analytics Accelerator z/os: Recognized leader in Security, Availability, & Recoverability for OLTP Supported by DB2 9 and above IBM DB2 Analytics Acelerator Netezza: Recognized leader in cost-effective high speed deep analytics Slide 66 of 80

68 Creating the Hybrid Data Server Combine DB2 for z/os with Netezza to provide an industry exclusive Data Mart Data Mart Data Mart Data Mart Consolidation Best in OLTP and Transactional Analytics Industry recognized leader in mission critical transaction systems Transaction Processing Systems (OLTP) Best in Complex Analytics Proven appliance leader in high speed analytic systems Transactional Analytics Complex Analytics z/os: Recognized leader in transactional workloads with security, availability and recoverability DB2 Analytics Accelerator Powered by Netezza: Recognized leader in cost-effective high speed deep analytics Together: Destroying the myth that transactional and decision support workloads have to be on separate platforms Best in Consolidation Unprecedented mixed workload flexibility and virtualization providing the most options for cost effective consolidation Slide 67 of 80

69 Workload Assessment Query Results Slide 68 of 80

70 International Securities Company Customer Goals utilizing IBM DB2 Analytics Accelerator and Smart Analtyics System 9700 The customer is currently utilizing Sybase for their OLTP and Data Warehouse environments and moving the OLTP environment to DB2 for z/os. The customer conducted a benchmark to determine if DB2 for z/os with the accelerator is a viable platform for data warehousing. Customer would like to reduce MIPs, Software Costs and the Total Cost of Ownership with the 9700 and Accelerator Customer Statistics Data Warehouse is approximately 12 TB accelerated into the Accelerator 100 Simple queries and 112 complex queries were tested in the benchmark IBM DB2 Analytics Accelerator (Netezza ) Day 1-2: Hardware installation and configuration Day 3: Created and loaded tables in the Accelerator and executed first queries 212 queries were executed sequentially (9700 only and then with the Accelerator) Slide 69 of 80

71 Query Details Several Ran Sub-Second Up to 358x Faster! Avoided CPU Consumption Avoided Redirecting Slide 70 of 80

72 Here s My Top 10 Best Practices Partitioning 1 Parallelism 2 Data 5 compression Index 6 Compression (maybe not) Star 3 (Snowflake) Schema Query 4 Processing Manage 7 statistics Maintain 8 accounting and statistical data Are 9 backups necessary DSNZPARMs 10 DFSORT 12 Buffer 11 Pools Slide 71 of 80

73 Shameless Self Promotion Please Visit My DB2 for z/os Blog QR code points directly to my blog Slide 72 of 80

74 Optimizing DB2 Queries with IBM DB2 Analytics Accelerator for z/os For Twitter use hashtags #zdwdb2 The Accelerator a marriage between System z and Netezza technology to accelerate complex queries in a DB2 for z/os highly secure and available environment. Superior performance and scalability with rapid appliance deployment provide an ideal solution for complex analysis. Provides a broad understanding of the IBM DB2 Analytics Accelerator architecture Documents the installation steps of this solution in an existing DB2 10 for z/os environment. SG Slide 73 of 80

75 Getting Started with the IBM Smart Analytics System 9600 For Twitter use hashtags #zdwdb2 A single, end-to-end business analytics solution to accelerate data warehousing and business intelligence initiatives Provides integrated hardware, software, and services that enable enterprise customers to quickly and cost-effectively deploy businesschanging analytics across their organizations SG Slide 74 of 80

76 50TB System z Data Warehouse Benchmark Poughkeepsie Lab / Silicon Valley Lab joint effort Deliver proof points of System z scalability in Business Intelligence environment Establish capability of System z to scale to larger volumes Develop best practices of managing large data warehouses Drive unique value of System z Workload Manager: capabilities to manage mixed workloads Operational BI: large volumes of users, smaller queries ziip: lower cost BI solution Data compression: minimal overhead with hardware compression SG Slide 75 of 80

77 Co-locating Transactional and Data Warehouse Workloads For Twitter use hashtags #zdwdb2 Poughkeepsie Lab / Silicon Valley Lab joint effort Managing analytic and traditional workload best practices Using workload manager (WLM) to manage mixed workloads OLAP vs data warehouse workloads Data sharing vs non data sharing considerations Single vs multiple LPAR configurations Implementation considerations Benchmark results SG Slide 76 of 80

78 Other Significant System z Data Warehouse Redbooks For Twitter use hashtags #zdwdb2 SG Enterprise Data Warehousing with DB2 9 for z/os SG InfoSphere Warehouse: A Robust Infrastructure for Business Intelligence REDP-4606 Using IBM System z As the Foundation for Your Information Management Architecture Slide 77 of 80

79 More Significant System z Data Warehouse Redbooks For Twitter use hashtags #zdwdb2 REDP-3927 Workload Management for DB2 Data SG Best Practices for SAP BI using DB2 9 for z/os REDP-4668 IBM zenterprise BladeCenter Extension Slide 78 of 80

80 Thank You For Attending! Slide 79 of 80

81 Willie Favero DB2 SME Data Warehousing for System z Swat Team IBM Silicon Valley Laboratory Slide 80 of 80