Performance tuning in SDS VA with a remote DB2 DB

Transcription

1 Performance tuning in SDS VA with a remote DB2 DB IBM SECURITY SUPPORT OPEN MIC PRESENTATION Ramamohan T Reddy - Senior Software Engineer / L2 Team Tech Lead - Directory Support Team Dave Bachmann - STSM / Performance Guru, IBM Security Systems performance Brook Heimbaugh - L3 Team Lead / Tech Lead - Directory Support Team Juan Quema - Software Engineer / L2 Team Member - Directory Support Team Tuesday, 18 April 2017

2 Agenda

3 Agenda SDS Virtual Appliance - Remote Database - Previous Presentations Directory Server Components - Tuning Perspective LDAP Server Caches DB2 Memory Management and Buffer Pools LDAP Server worker threads and DB2 connections Data import / load considerations Data Organization Data Optimization Indexing the attributes used in search filters Additional DB2 tuning Additional performance tuning resources Useful Links Questions for the Panel 3 IBM Security

4 SDS Virtual Appliance - Remote Database - Previous Presentations

5 SDS VA - Remote Database - Previous Presentations Configuring SDS Virtual Appliance with a remote DB2 database Securing communication between SDS VA and its remote DB2 DB Upgrade from SDS 6.4 to SDS 8.01 VA to use existing remote DB2 DB 5 IBM Security

6 Directory Server Components - Tuning Perspective

7 Network Network Directory Server Components - Tuning Perspective Main goal of this presentation is to understand and perform several performance tuning aspects of SDS VA Directory Server with Remote DB. Performance tuning is iterative and periodic task. SDS VA Remote DB2 System LDAP Server DB2 Server Client Applications Filter Cache Entry Cache STMM Instance Memory Database Memory IBM Default Buffer Pool IBMDEFAULTBP (Page Size: 4K) Disks Disks ACL Cache LDAP Buffer Pool LDAPBP (Page Size: 32K) 7 IBM Security

8 LDAP Server Caches

9 LDAP Server Caches Faster data retrieval - Benefits Search operations. ACL Cache: Stores info about access permissions for recently queried entries Entry Cache: Stores ldap entries retrieved from DB2 upon first search/query. Returns the same entry, if unchanged for subsequent searches. Filter Cache: Stores entry IDs that match specific search filters that were previously resolved in DB2 Attribute Cache: Stores configured attributes and their values. Gets used if the filter/entry caches doesn t resolve the results before going to DB2. This feature is deprecated and disabled by default. Group Members Cache: Extension of Entry Cache, stores member and uniquemember attributes. By default 25 groups with 25K members (or less) can be cached Recommendations for dynamic ldap data update environments Default cache settings are better for almost 95% customers. Higher cache sizes may degrade performance due to cache invalidations LDAP caches can be configured either using Web Admin Tool or idsldapmodify with proper ldif. 9 IBM Security

10 LDAP Server Caches Monitor cache utilization using cn=monitor search: sdsva > sds client_tools idsldapsearch -h server -p port -s base Cache related parameters of interest from cn=monitor search: -b cn=monitor objectclass=* filter_cache_size: The maximum number of filters that are supported by the cache. filter_cache_current: The number of filters currently in the cache. filter_cache_hit: The number of filters that are retrieved from the cache rather than being resolved in DB2. filter_cache_miss: The number of filters that were not found in the cache that then needed to be resolved by DB2. filter_cache_bypass_limit: Search filters that return more entries than this limit are not cached. entry_cache_size: The maximum number of entries that are supported by the cache. entry_cache_current: The number of entries currently in the cache. entry_cache_hit: The number of entries that were retrieved from the cache. entry_cache_miss: The number of entries that were not found in cache that then needed to be retrieved from DB2. acl_cache: A boolean value that indicates whether the ACL cache is active (TRUE) or inactive (FALSE). acl_cache_size: The maximum number of entries in the ACL cache. idsmonitor tool is now available on SDSVA Provides repeated cn=monitor searches at regular intervals of time. sdsva > sds client_tools idsmonitor -H <Host IP> -p <port> -D cn=root -w <passwd> -d <delay_in_secs> -o <outputfile> 10 IBM Security

11 DB2 Memory Management and Buffer Pools

12 DB2 Memory Management and Buffer Pools DB by default uses STMM (Self Tuning Memory Manager) Configuration update is required for migrated DB2 instances / databases to enable self tuning memory. The following DB2 memory consumers are automatically tuned: database_memory Database shared memory size buffer pools Number of pages in IBMDEFAULTBP and LDAPBP locklist Maximum storage for lock list maxlocks Maximum percent of lock list before escalation pckcachesz Package cache size sheapthres_shr Sort heap threshold for shared sorts sortheap Sort heap size Most of the System Memory should be allocated to DB2 memory. If DB2 memory parameters (Instance_memory or database_memory) and/or DB2 buffer pools are manually configured, the total DB2 memory consumption should be lower than system memory to avoid OS paging. Recommendation: System memory should be planned in accordance to the size of the database. Prefer automatic tuning (STMM) of DB2 memory and buffer pools 12 IBM Security

13 DB2 Memory Management and Buffer Pools Buffer pool utilization can be monitored using perfanalyze_database script from Performance tuning scripts. # su - db2inst1 $ db2 connect to LDAPDB $ db2 update monitor switches using bufferpool on sort on table on statement on $ db2 reset monitor all < Wait for your work load / searches etc > uow on lock on $ db2 get snapshot for all on LDAPDB > snap_1.out $./tuning_scripts/perfanalyze_database.pl -i snap_2.out... Bufferpool: IBMDEFAULTBP... Current size: pages Hit ratios (the higher the better) Data: % Index: % Total: % NOTICE: Total Hit Ratio should be >= 90% (the higher the better) Increase the size of the bufferpool until this value approaches 99%. 13 IBM Security

14 LDAP Server worker threads and DB2 connections

15 LDAP Server worker threads and DB2 connections To handle concurrent load from client applications, increase the number of ibmslapd worker threads. Number of worker threads is controlled by ibm-slapddbconnections attribute value under cn=directory, cn=rdbm Backends, cn=ibm Directory, cn=schemas, cn=configuration ibm-slapddbconnections also controls number of DB2 connections from ibmslapd to DB2 server In DB2 server process there will be as many agent threads as the number of client connections Increasing ibm-slapddbconnections requires more CPU / Memory resources Applies for both LDAP server as well as DB2 server. Monitoring ibmslapd worker thread utilization: sdsva > sds client_tools idsldapsearch -h server -p port -s base CN=MONITOR livethreads=3 available_workers=12 Recommendations: -b cn=monitor objectclass=* available_workers livethreads Increasing ibm-slapddbconnections may NOT improve performance in all cases Evaluate other performance bottlenecks using audit log and db2 snap shot analysis Do not configure very high number of ibm-slapddbconnections with out enough CPU / Memory resources 15 IBM Security

16 Data import / load considerations

17 Data import / load considerations For online update when ibmslapd (directory server) process is running: sdsva > sds client_tools idsldapadd -h sdsva -p 389 -D cn=root -w pw -i smalldata.ldif For offline small ldif data import / load use idsldif2db after uploading ldif file to virtual appliance sdsva > sds server_tools idsldif2db -I sdsinst1 -i smalldata.ldif For larger ldif data import / load use idsbulkload after uploading ldif file to virtual appliance. Use idsbulkload command to parse ldif file and create a tar.gz file containing table specific load files: sdsva > sds server_tools idsbulkload -I sdsinst1 -i data.ldif -a parseonly -Y... GLPBLK073I Bulkload completed. The file 'remote_bulkload.tar.gz' can be downloaded and viewed under CustomOut folder in LMI. On remote DB2 server system - Download remote_bulkload.tar.gz file and extract: ==> su - db2inst1 $ mkdir remote_bulkload_data $ cd remote_bulkload_data/ $ mv../remote_bulkload.tar.gz. $ gunzip -c remote_bulkload.tar.gz tar -xf - From the same folder review the script ldapdb_remote.sh (for AIX/Linux) [ldapdb_remote.bat for windows] Edit script and update the DB2INSTANCE variable to the correct db2 instance name and run the script as below $./ldapdb_remote.sh db2inst1 passwd 17 IBM Security

18 Data Organization

19 Data Organization High volume of insert, update & delete activity causes data fragmentation in tables and indexes. Reorganization compacts the data, reclaims disk space, improves data access Use reorgchk to determine the need for table / index reorganization # su - db2inst1 $ db2 connect to LDAPDB $ db2 reorgchk update statistics on table all > reorgchk.out Based on reorgchk.out file contents, all tables/indexes with asterisk requires reorg # su - db2inst1 $ db2 connect to LDAPDB $ db2 reorg table <table_name> $ db2 reorg indexes all for table <table_name> Recommendations: Reorganize all the attribute tables that you want to use in searches Perform Reorgchk periodically and based on that output perform Reorg on required tables/indexes Table level reorg also performs index organization Post reorg, perform runstats to take advantage of reorganized tables/indexes Reorg must be performed when ibmslapd is stopped Reorg on large tables may take long time, plan accordingly, perform reorg in batches at available maintenance times Use perftune_reorg script from Performance tuning scripts, to perform reorg on all / specific tables $./tuning_scripts/perftune_reorg.sh LDAPDB SDSINST1 [Table1 Table2...] 19 IBM Security

20 Data Optimization

21 Data Optimization The DB2 manager maintains a set of statistics to represent distributed data Optimization includes runstats command to update these statistics to improve query performance. reorgchk command may be used to update statistics, in the same way the runstats was performed last time. Use runstats to update statistics on a given table. # su - db2inst1 $ db2 connect to LDAPDB $ db2 runstats on table <tablename> on all columns with distribution and detailed indexes all Recommendations: In SDS v8.0.1.* env, with runstats, use distribution and detailed indexes all Perform runstats periodically (once in a day / week based on update rate). Perform runstats during a low activity time frame, Manual runstats is preferred over the DB2 auto runstats. Remove LDAP_MAXCARD and IBMSLAPD_USE_SELECTIVITY lines from ibmslapd.conf (restart ibmslapd) Benefit: Uses latest performance model of using actual ancestor/parent eid values with sub/one scope related queries. Use perftune_runstats script from Performance tuning scripts, to perform runstats on all / specific tables $./tuning_scripts/perftune_runstats.sh LDAPDB SDSINST1 21 IBM Security

22 Indexing the attributes used in search filters

23 Indexing the attributes used in search filters Indexing attributes used in search filters improves search performance Indexing an attribute may be performed using either Web Admin Tool or using idsldapmodify: Updates LDAP Server schema definition (ibmattributetypes) for the attribute Adds necessary indexes in DB2 layer for the corresponding attribute specific table Recommendations: Use ibmslapd.log GLPRDB091W messages to find a list of attributes that require indexing: GLPRDB091W Attribute <attributename> is not indexed but is used in searches 1000 times. Use audit log to determine the indexing rule for a given attribute based on filter: Attribute with filter such as "cn=john Doe requires EQUALITY indexing rule Attribute with filter such as sn>=doe requires ORDERING Attribute with filter such as "sn~=doe requires APPROX Attribute with filter such as "sn=mcd*" requires SUBSTR Attribute with filter such as "sn=*onald" requires REVERSE At the minimum EQUALITY is required on an attribute that s used in search filters Do not index attributes that are NOT used in search filters Perform runstats post indexing operation to take full advantage of the index Prefer to run the indexing operation during a service window when ibmslapd is not available to applications Use perfanalyze_indexes script from Performance tuning scripts, to analyze an audit log to get an ldif with index modifications $./tuning_scripts/perfanalyze_indexes.pl -d <ldap_schema_dir> -i audit.log -l indexupdate.ldif 23 IBM Security

24 Additional DB2 tuning

25 Additional DB2 tuning Add explicit/direct DB2 Index OBJECTCLASS_EOC Improves objectclass=<value> sub-filter performance $ db2 'create index schema_name.objectclass_eoc on schema_name.objectclass ("EID" ASC, "OBJECTCLASS" ASC) MINPCTUSED 10 ALLOW REVERSE SCANS' DB2 transaction log file size (LOGFILSIZ) - needs to be increased to accommodate large entry updates. $ db2 get database configuration for ldapdb grep LOGFILSIZ Log file size (4KB) (LOGFILSIZ) = 2000 $ db2 update database configuration for LOGFILSIZE using To enhance performance of NOT (!) filters such as (!(uid=admin)) use DB2 query compiler variable DB2_ANTIJOIN with EXTEND, the optimizer searches for opportunities to transform both NOT IN and NOT EXISTS subqueries into anti-joins. $ db2set DB2_ANTIJOIN=EXTEND Its important that attribute in search filter must have an index with up to date runstats. Set proper ulimits for DB2 instance user on Linux/Unix systems: Recommended ulimit values: data: unlimited nofile: fsize: unlimited Enable Row Compression - Requires DB2 Storage Optimization License (not provided by SDS) Use perftune_compress_tables.sh script from Performance tuning scripts, to enable row compression. 25 IBM Security

26 Additional performance tuning resources

27 Additional performance tuning resources Enable Audit Logging using Web Admin Tool / Command line idsldapmodify Use ISIM Performance Tuning Guide Use ISIM Performance Tuning Scripts Useful in analyzing and tuning Directory Server / DB2 in ISIM, ISAM and other SDS environments Refer ISIM_Performance_Scripts_Documentation pdf file inside the scripts tar.gz for details on each script. Resolving slow queries using the Directory Server audit log Resolving slow queries using DB2 snapshots Directory Server Performance Tuning Document in SDS knowledge Center 27 IBM Security

28 Useful Links

29 Useful Links SDS Recommended Fixes SDS Knowledge Center SDS Support Portal SDS Download Document SDS Known Issues DB Knowledge Center DB2 Recommended Fix Packs Ask a question in either ISDS -DW Answers Forum OR Directory Suite - DW Answers Forum Passport Advantage (full product downloads) Directory Server Support Lifecycle Sign up for My Notifications Follow IBM Security support on: 29 IBM Security

30 Questions for the Panel Now is your opportunity to ask questions of our panelists. To ask a question now: Raise your hand by clicking Raise Hand. The Raise Hand icon appears next to your name in the Attendees panel on the right in the WebEx Event. The host will announce your name and unmute your line. or Type your question in the box below Ask drop-down menu in the Q&A panel. Select All Panelists from the Ask drop-down-menu. Click Send. Your message is sent and appears in the Q&A panel To ask a question after this presentation: You are encouraged to participate in our dw Answers Forums: Ask a question in either ISDS -DW Answers Forum OR Directory Suite - DW Answers Forum 30 IBM Security

31 THANK YOU FOLLOW US ON: ibm.com/security securityintelligence.com youtube/user/ibmsecuritysolutions Copyright IBM Corporation All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and / or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM DOES NOT WARRANT THAT ANYSYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOUR ENTERPRISE IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.