IBM Security Guardium: Troubleshooting No Traffic Issues IBM SECURITY SUPPORT OPEN MIC To hear the WebEx audio, select an option in the Audio Connection dialog or by access the Communicate > Audio Connection menu option. To ask a question by voice, you must either Call In or have a microphone on your device. You will not hear sound until the host opens the audio line. For more information, visit: http://ibm.biz/webexoverview_supportopenmic July 18, 2017 NOTICE: BY PARTICIPATING IN THIS CALL, YOU GIVE YOUR IRREVOCABLE CONSENT TO IBM TO RECORD ANY STATEMENTS THAT YOU MAY MAKE DURING THE CALL, AS WELL AS TO IBM S USE OF SUCH RECORDING IN ANY AND ALL MEDIA, INCLUDING FOR VIDEO POSTINGS ON YOUTUBE. IF YOU OBJECT, PLEASE DO NOT CONNECT TO THIS CALL.
Panelists Presenter John Adams Guardium Support Moderator Andrew McCarl Knowledge Manager, IBM Security 2 IBM Security
Agenda Welcome and overview Where s my traffic? Collector or STAP? Reports and Policy STAP and Connections ATAP and Local Traffic Open discussion 3 IBM Security
Dude, where s my traffic?? 4 IBM Security
Could be almost anything Sniffer is down STAP not installed STAP process not running KTAP or WFP not running ATAP needed but not enabled or misconfigured Network issue Firewall blocking STAP ports Policy issue Report conditions or runtime parameters Sniffer parser issue Even aggregation issues! https://goo.gl/images/nugmbi 5 IBM Security
It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so. - Mark Twain 6 IBM Security
My Favorite Test Invalid Username! STAP or Collector? Local traffic, remote or both? Why? Login exceptions always captured, regardless of policy Can test the exact instance and exact node Bogus username is easy to find in a report Proves that traffic was captured OR. Almost proves traffic was not captured How? Add pre-defined report, Failed Login Attempts to GUI Have DBA log into the instance from a remote TCP client with an invalid username like TestRemote Repeat with a local / shmem connection and username TestLocal Run Failed Login Attempts for NOW to NOW -15 MINUTE and see which exceptions were captured. 7 IBM Security
My Favorite Test Invalid Username! 8 IBM Security
My Favorite Test Invalid Username! Similar to STAP Verification feature, but Verification has some caveats May require special configuration / datasources Only tests remote TCP traffic Basic verification doesn t let you choose the exact node Results! Remote Local Troubleshoot Yes Yes Reports, policy No No STAP, KTAP/WFP, network, firewalls Yes No ATAP (Oracle) No Yes ATAP (Sybase) or Windows needs reboot Knowing where to start saves you time! 9 IBM Security
Local and Remote Captured Troubleshooting Reports and Policy
Troubleshooting Reports and Policy Make a clone Simplify the conditions Eliminate tuples and DB_USER (just for testing!) Fields like DB_USER and SOURCE_PROGRAM have place-holder values which get replaced later. This affects how policy is applied Watch out for nested AND/OR conditions Clone groups and test a small number of members Check the Main Entity on your report Main entity determines what each row represents Cross-check with a different report Compare conditions Compare main entity Verify by using the Allow All policy Simple default policy with no rules Don t leave this on in Production, you ll fill the Collector! 11 IBM Security
Nothing Captured Troubleshooting STAP, KTAP, WFP, Network, Firewall
Troubleshooting the Sniffer Local Taps: Are all STAPs red or just some? If any STAPs are yellow or green, Snif is running. From CLI: Stop inspection-core Start inspection-core Check the Buffer Usage Report When was the last traffic received? Was TID stable or changing? (sniffer crash) If the Sniffer restarts without errors but all STAPs are red, troubleshoot the firewall next. 13 IBM Security
Buffer Usage Report 14 IBM Security
Local Taps 15 IBM Security
Troubleshooting Firewall and Network Issues From the host, run traceroute to the collector IP UNIX port 16016 or 16018 (TLS) Windows port 9500 or 9501 (TLS) Minimal hops, latency under 100ms Blocked? Work with your firewall team. Ports need to be open both ways When in doubt, use STAP debug or a packet sniffer Review the ports Technote! Guardium v10.0/10.1/10.1.2 and v9.0/9.1/9.5 Open Ports STAP debug: IBM MustGather: Collecting data for Guardium STAP 16 IBM Security
Troubleshooting UNIX STAP and KTAP Issues Is STAP installed? find / -n guard_tap.ini Many (but not all) flavors check /etc/inittab for utap Is the STAP process running? ps ef grep i tap ls lhtr <guardium dir>/modules/stap/current How many copies of guard_tap.ini do you have? *.err? *.bak? What are the timestamps? Is KTAP running? lsomd grep i ktap (Linux) genkex grep i ktap (AIX) modinfo grep i ktap (Solaris) Should match your STAP version! OK if you have two (old and current version) Check syslog 17 IBM Security
Troubleshooting Windows STAP and WFP Issues Is STAP installed? Windows Services Is the STAP process running? Windows Services Event Log Applications Is WFP running? C:\Users\Administrator>sc query wfpmonitor SERVICE_NAME: wfpmonitor TYPE : 1 KERNEL_DRIVER STATE : 4 RUNNING (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0 C:\Users\Administrator>sc query lhmonproxy [SC] EnumQueryServicesStatus:OpenService FAILED 1060: The specified service does not exist as an installed service. 18 IBM Security
Only Remote / Only Local Captured Troubleshooting ATAP
Simplified Architecture of Unix/Linux STAP 20 IBM Security
Troubleshooting ATAP Issues Was ATAP configured and activated? Non-GIM: <install_dir>/guard_stap/guardctl list-active GIM: <install_dir>/modules/atap/current/files/bin/guardctl list-active /guardctl activate Was the DB just upgraded? Before upgrading the database ATAP must be deactivated. Otherwise, manual intervention will be required. /guardctl deactivate Did you reboot? When in doubt, try it. Needed for Windows STAP in some cases. (Not related to ATAP) IBM Guardium - When to Restart, When to Reboot Did you authorize the database user? /guardctl is-user-authorized /guardctl authorize_user db2admin 21 IBM Security
Consider Using an Exit Library Instead Easier to maintain Performs the same function as ATAP. Leverage native capabilities of the database to capture traffic. Uses DBMS specific libraries which ship with STAP. Supports: DB2 Teradata 16.10 Informix 12.10.xC6 Documentation: Configuring DB2_EXIT to integrate with Guardium Unix STAP 22 IBM Security
When All Else Fails
Must-Gather for PMRs support must_gather sniffer_issues STAP diag IBM MustGather: Collecting data for Guardium STAP Invalid username test: Captured remote? Local? Both? One node missing? The whole cluster? PDF or CSV of the report where you see the issue. Description of the missing traffic. Any other troubleshooting you have already done. 24 IBM Security
Questions for the panel Now is your opportunity to ask questions of our panelists. To ask a question now: Raise your hand by clicking Raise Hand. The Raise Hand icon appears next to your name in the Attendees panel on the right in the WebEx Event. The host will announce your name and unmute your line. or Type a question in the box below the Ask drop-down menu in the Q&A panel. Select All Panelists from the Ask drop-down-menu. Click Send. Your message is sent and appears in the Q&A panel. To ask a question after this presentation: You are encouraged to participate in the dw Answers forum: <https://developer.ibm.com/answers/topics/tag.html> 25 IBM Security
Where do you get more information? Questions on this or other topics can be directed to the product forum: https://developer.ibm.com/answers/topics/guardium.html. More articles you can review: IBM MustGather: Collecting data for Guardium STAP http://www-01.ibm.com/support/docview.wss?uid=swg21606592 What to do if you receive Guardium no traffic alert http://www-01.ibm.com/support/docview.wss?uid=swg21699786 IBM Knowledge Center: Predefined Alerts https://www.ibm.com/support/knowledgecenter/en/ssmphh_10.1.0/com.ibm.guardium.doc.admin/adm/predefi ned_alerts.html Useful links: Get started with IBM Security Support IBM Support Portal Sign up for My Notifications Follow us: 26 IBM Security
Disclaimer Please Note: IBM s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. 27 IBM Security
THANK YOU FOLLOW US ON: ibm.com/security securityintelligence.com xforce.ibmcloud.com @ibmsecurity youtube/user/ibmsecuritysolutions www.securitylearningacademy.com Copyright IBM Corporation 2017. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM's current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM does not warrant that any systems, products or services are immune from, or will make your enterprise immune from, the malicious or illegal conduct of any party.