Exploring Memory in DB2 9.5 and Beyond

Exploring Memory in DB2 9.5 and Beyond Michael Cornish, Senior Software Analyst Advanced Problem Diagnostics Team IBM Toronto Lab mcornish@ca.ibm.com 2009 IBM Corporation

TOPICS Memory Architecture Changes for DB2 9.5/9.7 Memory Model Update Instance Memory STMM 2 2009 IBM Corporation

Memory Architecture Changes Why? Reduce knobs which must be configured No need to configure for memory which is absolute requirement Database Heap, Application Control Heap, Application Heap Focus on tuning performance-related memory areas (eg. bufferpools) Make memory allocation more dynamic Avoid recycling, Increase STMM tuning capability Made possible by: Moving away from 32-bit constraints (virtual process address space) Multithreaded model (reduce need for fixed shared memory areas) 3 2009 IBM Corporation

Memory Model Update DB2 9.5 Model FCM MEMORY (shared across all local partitions) ( portion billed to this partition s Instance Memory ) INSTANCE MEMORY (per partition) DBMS memory FMP memory db2start database activation Database A Memory APPL Memory Private Memory Database B Memory APPL Memory LOCAL LOCAL Communication Memory Communication Memory per local connection db2start 4 2009 IBM Corporation

Memory Model Update DB2 9.5 Model - DPF FCM MEMORY (shared across all local partitions) INSTANCE MEMORY (Partition 0) DBMS FMP INSTANCE MEMORY (Partition 1) DBMS FMP Database A Memory APPL Database B Memory APPL Database A Memory APPL Database B Memory APPL Private Memory Private Memory LOCAL LOCAL LOCAL LOCAL DPF 2 partitions 5 2009 IBM Corporation

Memory Model Update How to see it in DB2? db2pd dbptnmem (sysproc.admin_get_dbp_mem_usage) High-level, all areas, for monitoring Instance Memory usage includes allowance for configuration, not just actual system memory usage Mem Used includes Cached Individual Memory Consumers: Name Mem Used (KB) HWM Used (KB) Cached (KB) ======================================================== APPL-SAMPLE 160000 160000 158400 DBMS-mcornish 49536 49536 10240 FMP_RESOURCES 22528 22528 0 PRIVATE 11200 15104 832 LCL-p2617558 128 128 0 DB-SAMPLE 1052800 1052800 174840 FCM_RESOURCES 69760 69760 0 6 2009 IBM Corporation

Memory Model Update How to see it in DB2? Shared Memory: db2pd inst memsets db <database> -memsets Commited (Cmt) value is accurate measure of system memory usage More detailed, useful, no SQL Can map to OS resources (IPC shared segment IDs) Memory Set Name Id Size(Kb) Cmt(Kb) SAMPLE 361758884 148800 66176 (DB/Database Memory) AppCtl 623902804 160064 2496 (APPL Memory) App73 145752160 128 128 (Local Communication) DBMS 748683319 35712 9728 FMP 265289809 22592 22592 FCM 358613130 220544 139520 7 2009 IBM Corporation

Memory Model Update How to see it in DB2? Private memory: db2 get snapshot for database manager Committed private Memory (Bytes) = 15335424 db2pd dbptnmem (Used Cached) Name Mem Used (KB) HWM Used (KB) Cached (KB) ======================================================== PRIVATE 16320 17600 3904 - Greater reliance on operating system tools as private memory includes more than just DB2 s tracked allocations - ps, pmap, etc. 8 2009 IBM Corporation

Memory Model Update How to see it in OS? Shared ipcs a (UNIX) * virtual size only * pmap (Solaris, Linux) svmon (AIX) Private ps (except Solaris, Linux) pmap (Solaris, Linux) Other glance (HP) perfmon (Windows) 9 2009 IBM Corporation

Memory Model Update Private Memory FCM MEMORY (shared across all local partitions) ( portion billed to this partition s Instance Memory ) INSTANCE MEMORY (per partition) DBMS memory FMP memory Database A Memory APPL Memory Private Memory Database B Memory APPL Memory LOCAL LOCAL Communication Memory Communication Memory db2start 10 2009 IBM Corporation

Memory Model Update Private Memory Single large memory area per partition, allocated as needed At OS level, shared by all DB2 threads (EDUs/Agents) At DB2 level, most memory is private to a thread, i.e. DB2 manages memory in thread-specific memory pools Non-configurable except private sort ( SHEAPTHRES > 0 ) Per-thread memory footprint is 0.5-1 MB 11 2009 IBM Corporation

Memory Model Update Private Memory Consolidate, simplify Private Memory, eliminate bloat Active Agent DB2 9.1 DB2 9.5 Active Agent Active Agent Statement Heap Application Heap Statistics Heap APPLICATION MEMORY (APPL_MEMORY) Unused Commited Inactive Agent Cached Unused Private Memory Inactive Agent Agent/ Thread 1 PRIVATE MEMORY Agent/ Thread 2 Agent/ Thread 3 12 2009 IBM Corporation

Memory Model Update Database Memory FCM MEMORY (shared across all local partitions) INSTANCE MEMORY (per partition) DBMS memory FMP memory database activation Database A Memory APPL Memory Private Memory Database B Memory APPL Memory LOCAL LOCAL Communication Memory Communication Memory 13 2009 IBM Corporation

Memory Model Update Database Memory One Database Memory area per database, per partition Configurable via DATABASE_MEMORY SETTINGS: <value>, AUTOMATIC, COMPUTED AUTOMATIC, COMPUTED same behaviour with STMM OFF Size based on bufferpools, dbheap, utility heap, etc. AUTOMATIC allows STMM tuning if STMM ON **AUTOMATIC now available on ALL UNIX** **ONLY memory area tuned by STMM** 14 2009 IBM Corporation

DB2 9.5 / Linux AUTOMATIC Restriction Can t set both Database Memory and Instance Memory to AUTOMATIC Applies to Linux levels before SLES 10 SP1, RHEL 5 Lacking good system statistics, can t rely on STMM to keep DB2 s memory usage within system capacity Doesn t affect DB2 9.7 due to higher Linux level prereq Need one of: DATABASE_MEMORY = COMPUTED or <value> INSTANCE_MEMORY = <value> 15 2009 IBM Corporation

Memory Model Update Database Memory OVERFLOW Package Cache Package Cache Catalog Cache Utility Heap Database Heap ~20% Overflow Reserved LOCKLIST Shared Sortheap BUFFERPOOLS DATABASE MEMORY 16 2009 IBM Corporation

Memory Model Update Application Memory FCM MEMORY (shared across all local partitions) ( portion billed to this partition s Instance Memory ) INSTANCE MEMORY (per partition) DBMS memory FMP memory database activation Database A Memory APPL Memory Private Memory Database B Memory APPL Memory LOCAL LOCAL Communication Memory Communication Memory 17 2009 IBM Corporation

Memory Model Update Application Memory One Application Memory area per database, per partition Configurable via APPL_MEMORY Options: Specific Value, AUTOMATIC Should not need to be tuned (leave at AUTOMATIC) Volatile memory area which is efficiently self-managed Most memory is a strict requirement as opposed to performance/cache Memory usage dependent on number of connections, activity 18 2009 IBM Corporation

Memory Model Update Application Memory Statistics Heap STAT_HEAP_SZ Application Group Shared Heap (persistent,non-configurable) APPLICATION MEMORY (APPL_MEMORY) Statement Heap STMTHEAP Statement Heap STMTHEAP Statement Heap STMTHEAP Application Heap APPLHEAPSZ Application Heap APPLHEAPSZ Application Heap APPLHEAPSZ.. All Memory Heaps/Pools grow as needed, release memory when no longer used 19 2009 IBM Corporation

Memory Model Update Application Memory No more Application Groups (DPF, Intra-parallel, Concentrator) No more complex configuration, memory bloat DB2 9.1 DB2 9.5 APP_CTL_HEAP_SZ =? GROUPHEAP_RATIO =? APPGROUP_MEM_SZ =? APPL_MEMORY = AUTOMATIC Appgroup 1 Appgroup 2 Appgroup 3 APPLICATION MEMORY Appgroup 4 Appgroup 5 20 2009 IBM Corporation

Memory Model Update DB2 9.5 Model Questions? FCM MEMORY (shared across all local partitions) ( portion billed to this partition s Instance Memory ) INSTANCE MEMORY (per partition) DBMS memory FMP memory db2start database activation Database A Memory APPL Memory Private Memory Database B Memory APPL Memory LOCAL LOCAL Communication Memory Communication Memory per local connection db2start 21 2009 IBM Corporation

Memory Architecture Changes - Dynamic Shared Memory STMM tuning of Database Memory on all UNIX already available in version 9.1 on AIX and Windows No need to configure many memory parameters No fixed shared memory size (remember 32-bit?) On increase, shared memory segment added On decrease, underlying system memory released Virtual process allocation/mapping is persistent 22 2009 IBM Corporation

Memory Architecture Changes - Dynamic Shared Memory New Allocation Requirements decrease Memory Requirements Increase Uncommitted Memory Committed Memory (Backed by system i.e. RAM) Committed Memory Committed Memory Decommitted memory Decommitted memory Committed Memory Decommit frees underlying system memory, virtual allocation remains 23 2009 IBM Corporation

Memory Architecture Changes - MORE AUTOMATIC! Database Manager Memory MON_HEAP_SZ (Monitor Heap) Database Memory DBHEAP (Database Heap) Application memory APPL_MEMORY itself (formerly APPGROUP_MEM_SZ) APPLHEAPSZ (Application Heap) - includes App Control Heap STAT_HEAP_SZ (Statistics Heap) STMTHEAP (Statement Heap) *no change on migration* 24 2009 IBM Corporation

Memory Architecture Changes - MORE AUTOMATIC! Meaning of AUTOMATIC for new settings: Unlimited growth (up to any higher-level limit) OK as these have strict requirements not for performance STMTHEAP: dynamic join limited -> greedy join unlimited Other meanings of AUTOMATIC STMM Automatically tune Database Memory areas Automatically calculated/computed setting FCM grow and shrink based on demand 25 2009 IBM Corporation

Memory Architecture Changes AGENT_STACK_SZ Stacks managed separately from main DB2 memory model Fixed stack size required for multithreaded model Allocated as needed (up to limit), released when agent terminates Overall stack usage is insignificant Only specific activity, eg. statement compilation, may require large stack If not large enough, SQL0973 returned, stack size must be increased (not dynamic) SQL0973 generally treated as product defect need to reduce stack usage 26 2009 IBM Corporation

BEST PRACTICES New Memory Settings APPL_MEMORY = AUTOMATIC APPLHEAPSZ = AUTOMATIC STAT_HEAP_SZ = AUTOMATIC **STMTHEAP = <value> Improved AUTOMATIC behaviour change coming DB2 9.5 Fix Pack (IC68794), DB2 9.7 Fix Pack 3 (IC69070) DATABASE_MEMORY = AUTOMATIC (other options) DBHEAP = AUTOMATIC (<value>) Value is reserved use default value i.e. AUTOMATIC (1200) for small database DBMS - Database Manager Memory (not configurable) MON_HEAP_SZ = AUTOMATIC (<value>) Value is reserved use default value i.e. 90 for small database AGENT_STACK_SZ Default 1024 for OLTP, 4096 for DSS/Warehouse, Mixed 27 2009 IBM Corporation

INSTANCE MEMORY What is it? FCM MEMORY (shared across all local partitions) ( portion billed to this partition s Instance Memory ) INSTANCE MEMORY (per partition) DBMS memory FMP memory db2start database activation Database A Memory APPL Memory Private Memory Database B Memory APPL Memory LOCAL LOCAL Communication Memory Communication Memory per local connection db2start 28 2009 IBM Corporation

Instance Memory What is it? New accounting method Tracks main DB2 memory areas (Database, Application ) Accounts for both usage and configuration (eg. reservations) Memory Manager Gives permission for memory areas to grow update db cfg using Database_Memory <value> 29 2009 IBM Corporation

Instance Memory What is it? db2pd dbptnmem Database Partition Memory Controller Statistics Controller Automatic: Y Memory Limit: 3488912 KB Current usage: 453312 KB HWM usage: 454720 KB Cached memory: 187200 KB Individual Memory Consumers: Name Mem Used (KB) HWM Used (KB) Cached (KB) ======================================================== APPL-SAMPLE 160000 160000 158400 DBMS-mcornish 35712 35712 320 FMP_RESOURCES 22528 22528 0 PRIVATE 16320 17600 3904 FCM_RESOURCES 69760 69760 0 DB-SAMPLE 148736 148736 24576 30 2009 IBM Corporation

Instance Memory What is it? OVERFLOW (cached) Cached Package Cache Package Cache Utility Overflow (buffer) Reserved Used Catalog Cache Heap Database Heap LOCKLIST Shared Sortheap BUFFERPOOLS DATABASE MEMORY RESERVED 31 2009 IBM Corporation

Instance Memory What is it? Used = Current size (In use by pools/heaps + reserved for configuration + overflow) Cached = Reducible, Unused overflow (DB, DBMS, FCM) ( can be reduced to make room elsewhere or reduce memory usage ) Individual Memory Consumers: Name Mem Used (KB) HWM Used (KB) Cached (KB) ======================================================== APPL-SAMPLE 160000 160000 158400 DBMS-mcornish 49536 49536 10240 FMP_RESOURCES 22528 22528 0 PRIVATE 11200 15104 832 LCL-p2617558 128 128 0 DB-SAMPLE 1052800 1052800 174840 FCM_RESOURCES 69760 69760 0 32 2009 IBM Corporation

Instance Memory INSTANCE_MEMORY INSTANCE_MEMORY (Database Manager configuration) AUTOMATIC or <value> AUTOMATIC computed value = 75-95% of RAM / # local partitions limit enforcement removed in DB2 9.5 FP5, DB2 9.7 FP1 except license memory limit still enforced <value> - Limit is enforced monitoring/sizing of Instance Memory important - Provides a target for STMM (ignores OS) - DPF Limitation single value for all partitions 33 2009 IBM Corporation

Free Used (includes Cached/ Reserved) Instance Memory vs. System Memory committed Database Memory committed Application Memory INSTANCE_MEMORY = 48GB OS & other Apps Instance Memory (Uncommited) Instance Memory (Committed) 64GB RAM Instance Memory Usage Accounting Difference System Memory Usage committed Private Memory 34 2009 IBM Corporation

Instance Memory - Summary Provides an overall limit on DB2 memory consumption Some areas not included (stack, memory from other software layers) Greater control with multiple instances Ensure safety room (taken care of by STMM) Free = Limit Used + Cached 3488912 453312 + 187200 = 1222800 1222800/3488912 = 35% db2pd -dbptnmem: Memory Limit: 3488912 KB Current usage: 2453312 KB HWM usage: 2454720 KB Cached memory: 187200 KB 35 2009 IBM Corporation

STMM DB2 s Self-Tuning Memory Manager Tunes most Performance memory Bufferpools, Locklist, Sort, Package Cache Targets optimal memory consumption OS memory statistics, performance statistics Similar to Operating System behavior OS uses available memory for file cache STMM makes use of available memory if sufficient benefit Can tune multiple databases within and across instances STMM controllers for each database compare benefits Equilibrium is reached when each database is getting the same marginal benefit from memory 36 2009 IBM Corporation

STMM : DATABASE_MEMORY = AUTOMATIC INSTANCE_MEMORY = AUTOMATIC (no limit, STMM monitors OS) DATABASE Untouchable memory kernel, other Apps, minimum file MEMORY cache Free System Memory Target More Benefit STMM DATABASE MEMORY Less Benefit Sorts & Hash Joins DB2 memory usage (outside of Database Memory) 37 2009 IBM Corporation

STMM : DATABASE_MEMORY = AUTOMATIC INSTANCE_MEMORY = AUTOMATIC (no limit, STMM monitors OS) DATABASE MEMORY Untouchable memory kernel, other Apps, minimum file cache If increase, STMM compensates Free System Memory Target DATABASE MEMORY More Benefit STMM Less Benefit Sorts & Hash Joins If increase, STMM compensates DB2 memory usage (outside of Database Memory) 38 2009 IBM Corporation

STMM : DATABASE_MEMORY = AUTOMATIC DATABASE MEMORY INSTANCE_MEMORY = <value> (STMM monitors Instance Memory) Free Instance Memory Target More Benefit STMM DATABASE MEMORY Less Benefit Sorts & Hash Joins DB2 memory usage outside of Database Memory 39 2009 IBM Corporation

STMM Multiple Databases System Memory Note NO Instance Boundaries Kernel, Other Application Memory, Minimum File Cache STMM Database A Free System Memory Target STMM Database B STMM Database C STMM Database D 40 2009 IBM Corporation

STMM Multiple Databases System Memory Kernel, Other Application Memory, Minimum File Cache STMM Database A Increased Benefit Free System Memory Target STMM Database B Decrease Database C STMM STMM Database D 41 2009 IBM Corporation

STMM Multiple Databases Instance Memory INSTANCE_MEMORY = <value> STMM Database A INSTANCE MEMORY LIMIT Free Instance Memory Target STMM Database B STMM Database C STMM Database D 42 2009 IBM Corporation

STMM Multiple Databases - Hybrid Kernel, Other Application Memory, Minimum File Cache Instance 1 (unlimited) Database A STMM Instance 3 (limited) STMM Database C Free System Memory Target Free Instance Memory Target Instance 2 (unlimited) Database B STMM STMM Database D 43 2009 IBM Corporation

STMM Miscellaneous Configuration SELF_TUNING_MEM is the master switch Configuration parameters and buffer pool sizes should be set to AUTOMATIC to enable tuning Persistence of changes All configuration changes are written to disk. If you start and stop the database all changes are preserved. Logging All configuration changes are logged to db2diag.log. Internal tuning information is logged to STMM log. Best used with the STMM log parser -http://tinyurl.com/y258lzx 44 2009 IBM Corporation

BEST PRACTICES Summary AUTOMATIC, AUTOMATIC, AUTOMATIC Put Instance Memory Cap on Databases which may benefit from more memory but where performance is not important Rogue queries may penalize other Databases competing for same resources STMM treats all databases equal Extremely volatile databases may require some fixed values for Performance Memory STMM is reactive, major workload changes need time to achieve optimal performance 45 2009 IBM Corporation

Exploring Memory in DB2 9.5 and Beyond? 46 2009 IBM Corporation