InterConnect Access Manager Performance Tuning Diagnostic Techniques. Nick Lloyd ISAM Level II Support 1 2/20/2017

Transcription

1 Access Manager Performance Tuning Diagnostic Techniques InterConnect 2017 Nick Lloyd ISAM Level II Support 1 2/20/2017

2 Please note IBM s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. 2 2/20/2017

3 Agenda Typical Deployment Pattern Trace Points Where should I start? Trace method for each point Common Settings Request log parsing tool 3 2/20/2017

4 Typical ISAM Deployment 4 2/20/2017 F I R E W A L L F I R E W A L L L O A D B A L A N C E R DMZ Reverse Proxy DSC or Failover Cookie AAC / Federation DMZ Reverse Proxy DSC or Failover Cookie AAC / Federation Primary Policy Server Authorization Servers Remote DSC Remote AAC / Federation EAI / LRR Application Server Junction Application Servers L D A P A D D B R E S T A p p D B L O A D B A L A N C E R

5 Common communications flow Internet User <-> Firewall <-> LB <-> Reverse Proxy <-> Firewall <-> Junction Application Servers Authentication (LDAP / AAC / Federation / EAI) Application Data There are 7 delta points, really 14 when you consider both ways. 5 2/20/2017

6 What has Support seen as Root Cause? We have seen it all!!!! Poor settings, across the board. Down-level versions at all points. Firmware and Software. Improper Licensing. Bad LDAP design. Too much auditing. Bad network design. Bad routing. LB Round-robin. Chewed up cable. 6 2/20/2017

7 Diagnostic Methodology 7 2/20/2017

8 Access frontend directly without LB. Is there a noticeable improvement? Problem with LB or network path via LB. Access backend servers directly without a LB. Is there a noticeable improvement? Problem with LB or network path via LB. Use ISAM junction LB or debug. Access with or without LB is fine. Now what? Backend Infrastructure. Contact application team. What do their request logs show? Are response times within criteria? No, they have some tuning to do. Yes, could be network. Time to capture pdweb.debug or use request.log %F to get more details. Set request-log-format = %F %r in conf file. Can easily parse with sort command. If not possible to set as such use pdweb.debug. Set pdweb-debug. New usec debug output. Use this to get TCP details. Slow response from backend. Investigate backend infrastructure. No issues here then move onto frontend. Frontend Infrastructure. Collect on appliance. Leave interface blank and filter of TCP. Collect on client. Follow same network diagnostic steps as backend. No issues here then look at ISAM Appliance and Reverse Proxy settings. ISAM Appliance and Reverse Proxy Settings. Appliance TCP Tunings Worker Threads. LDAP authentications. Auditing. 8 2/20/2017

9 Backend Diagnostics 9 2/20/2017

10 Backend network trace diagnostic steps Collect on appliance. Leave interface blank and filter of tcp. Collect pdweb.debug. Collect on backend server. Five possibilities. WebSEAL to Backend communication (Network Infrastructure). Backend to Application time (TCP stack, App threads, etc.) Application performance. Application to Backend time. Backend to WebSEAL communication. 10 2/20/2017

11 Analyze pdweb.debug.log $ grep usec pdweb.debug.log awk '{ print $11 }' sort -rn ============= Might as well look at the worst one Find thread in pdweb.debug.log. Get the ephemeral port :06: :00I----- thread(258) PD ===> BackEnd Thread ; fd 21; local :59434; remote :8080^M GET /cgi-bin/should_be_fast.sh HTTP/1.1^M :06: :00I----- thread(258) PD <=== BackEnd Thread ; fd 21; local :59434; remote :8080^M :06: :00I----- thread(258) Processing time of the junction request: usec ( seconds!!!!) 11 2/20/2017

12 Network Trace Side-by-Side Search for tcp.port == We can tell the following: Initial TCP 3-way connection is fast. No issues getting there or getting back. Reverse Proxy sends HTTP request, gets there fast. No issues at application layer getting there. Reverse Proxy does not get response until 5 seconds later. But it got it fast once sent. No issues at application layer going back. If applications logs show fast response then that leaves tuning on the server. Perhaps TCP stack, application threads, etc. 12 2/20/2017

13 Frontend Diagnostics 13 2/20/2017

14 Frontend network trace diagnostic steps TCP trace on Appliance. Leave interface blank and filter of tcp. Collect pdweb.debug.log on Reverse Proxy. TCP trace on Client. Five possibilities Client to Appliance communication (Network Infrastructure). Appliance to Reverse Proxy (TCP stack, worker-threads, etc.). Reverse Proxy performance. Reverse Proxy to Appliance. Appliance to Client communication. Perform same analysis using ephemeral port from Client. 14 2/20/2017

15 Appliance and Reverse Proxy 15 2/20/2017

16 Check Common Settings Appliance TCP Tunings: Worker Threads. Non-EAI authentications using IRA trace. (Especially when LDAP is SSL). Auditing to Local Filesystem. Multiple instance can compete for the disk. Auditing to Syslog. If the syslog server goes down it is back to local cache file with multiple instances competing for the disk. Trying to tune the Auditing Event Pool and individual audit loggers 16 2/20/2017

17 TCP Tunings Support recommends the following settings: 17 2/20/2017

18 Worker Threads The default settings can be deceiving: worker-thread-hard-limit = 100 worker-thread-soft-limit = 90 When set like this and the threads hit the max, a 503 is not returned. Incoming requests are queued up until a work thread is available. This is by design and while users may report slowness returning a 503 immediately may not be the desired affect based on the type of transactions. Check the message log, you will only see a message about hitting the hard-limit: DPWWM1493E Junction '/jct' has reached it's worker thread hard limit. Increase work-threads. To monitor and get a better picture set to: worker-thread-hard-limit = 90 worker-thread-soft-limit = 75 When set like this once the max is hit a 503 is returned. The message log now contains both hard and soft messages: DPWWM1494W Junction '/jct' has reached it's worker thread soft limit. DPWWM1493E Junction '/jct' has reached it's worker thread hard limit. 18 2/20/2017

19 Slow Authentications Because EAI, AAC, and Federation all use a junction, issues with these services are normally found when investigating the backend infrastructure. Traditional LDAP authentication performance can be determined using the pdweb.authn statistic, pdweb.debug.log or an IRA trace. We have seen some setups which make complex use the Federated Directory feature. Adding in just a few increases the amount of searches that must be done. These next steps will help determine the slow points. 19 2/20/2017

20 Using pdweb.authn for Authentications If there are reports of slow authentications you can get an overall picture with this statistic component. Enable the component: 20 2/20/2017

21 Using pdweb.authn for Authentications Sample output: :19: :00I----- HPDRA0387I pdweb.authn statistics report :19: :00I----- pdweb.authn pass : :19: :00I----- pdweb.authn fail : :19: :00I----- pdweb.authn pwd exp : :19: :00I----- pdweb.authn max : :19: :00I----- pdweb.authn avg : :19: :00I----- pdweb.authn total : If there are large values and you need more details move onto the next steps. 21 2/20/2017

22 Using pdweb.debug for Authentications :56: :00I----- thread(4) Browser ===> PD POST /pkmslogin.form HTTP/ :56: :00I----- thread(4) Browser <=== PD HTTP/ Moved Temporarily Note the time difference. It took seconds. Use the same diagnostic approach. Start with LDAP logs, then take TCP traces. 22 2/20/2017

23 Using IRA trace for Authentications Enable the trace component pd.ivc.ira at level 9. This will give exact details. Create a pattern file for the awk command: $ cat auth-pattern.awk ira_entry.c:3361: ira_ldap_search_ext_s() base: ira_ldap.c:3765: ira_ldap_search_ext_s(): No ira_ldap.c:3785: ira_ldap_search_ext_s: Returning LDAP 23 2/20/2017

24 Using IRA trace for Authentications Run awk command: $ grep -f auth-pattern.awk pd.ivc.ira.log :59: :00I----- thread(12) trace.pd.ivc.ira:7 /home/webseal/ /src/pd/ivrgy/ira_entry.c:3361: ira_ldap_search_ext_s() base: O=FEDERATED scope: 2 filter: (&(uid=fuser)( (objectclass=eperson)(objectclass=person))) :59: :00I----- thread(12) trace.pd.ivc.ira:7 /home/webseal/ /src/pd/ivrgy/ira_ldap.c:3765: ira_ldap_search_ext_s(): No timeout - calling ldap_search_ext_s :59: :00I----- thread(12) trace.pd.ivc.ira:7 /home/webseal/ /src/pd/ivrgy/ira_ldap.c:3785: ira_ldap_search_ext_s: Returning LDAP rc x0 The complete algorithm actually involves more searches but the pattern will match. This analysis will give the exact details. This give you an idea of what to look for. This Federated LDAP is taking 6 seconds. Time to talk to the LDAP team. 24 2/20/2017

25 Audit Logging Often times auditing is the root cause of performance issues. Here is a typical setting, with buffer_size, flush_interval, and hi_water left at defaults. logcfg = audit.azn:file path=audit.azn.log,rollover_size= ,max_rollover_files=10,log_id=azn Here is a typical bad setting: logcfg = audit.azn:file path=audit.azn.log,rollover_size= ,max_rollover_files=10,log_id=azn, hi_water=1 logcfg = audit.authn:file path=audit.authn.log,rollover_size= ,max_rollover_files=10,log_id=authn, hi_water=1 logcfg = audit. path=audit.azn.log,rollover_size= ,max_rollover_files=10,log_id=http, hi_water=1 This hi_water setting is not optimal. Use it if you want to ensure that events get to disk as fast as possible. Doing so adversely impacts overall performance. The default settings for the EventPool and an agent are fine for most environments. This presentation does not get into fine tuning should it be required. This is about diagnostics. If you have ruled everything else out and suspect auditing then what is the next step to rule out auditing? Is there tracing? Yes. Should you use it? No, not really. Just comment out the auditing. If performance is better then time to look at tuning. 25 2/20/2017

26 Requests per Second Takes request logs and converts each second to a value. Uses PERL associative array to keep count per second Loops over array to generate gnuplot code to create a graph. Requires PERL and Gnuplot Among other things finds Load-balancers that are not balancing so well. 26 2/20/2017

27 Requests per Second Something seem off? One appliance running a little hot? Do not have any statistics enabled? Run this PERL script against a directory of request logs. $./websealrequest.pl default Processing directory: default Found: request.log Processing files default/request.log Success: request-per-second.gp & requests-per-second.dat run gnuplot requests-per-second.gp $ gnuplot requests-per-second.gp ## It will create requests-per-second.png. 27 2/20/2017

28 Resources Must-gather: Auditing Components: Statistic Components: Request log grapher: ostic%20tools 28 2/20/2017

29 Notices and disclaimers Copyright 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed as is without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply. the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and 29 2/20/2017

30 Notices and disclaimers continued Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera, Bluemix, Blueworks Live, CICS, Clearcase, Cognos, DOORS, Emptoris, Enterprise Document Management System, FASP, FileNet, Global Business Services, Global Technology Services, IBM ExperienceOne, IBM SmartCloud, IBM Social Business, Information on Demand, ILOG, Maximo, MQIntegrator, MQSeries, Netcool, OMEGAMON, OpenPower, PureAnalytics, PureApplication, purecluster, PureCoverage, PureData, PureExperience, PureFlex, purequery, purescale, PureSystems, QRadar, Rational, Rhapsody, Smarter Commerce, SoDA, SPSS, Sterling Commerce, StoredIQ, Tealeaf, Tivoli Trusteer, Unica, urban{code}, Watson, WebSphere, Worklight, X-Force and System z Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: /20/2017

31 31 2/20/2017 InterConnect 2017