IBM BigFix Relays Part 2

Transcription

1 IBM BigFix Relays Part 2 IBM SECURITY SUPPORT OPEN MIC December 17, 2015 NOTICE: BY PARTICIPATING IN THIS CALL, YOU GIVE YOUR IRREVOCABLE CONSENT TO IBM TO RECORD ANY STATEMENTS THAT YOU MAY MAKE DURING THE CALL, AS WELL AS TO IBM S USE OF SUCH RECORDING IN ANY AND ALL MEDIA, INCLUDING FOR VIDEO POSTINGS ON YOUTUBE. IF YOU OBJECT, PLEASE DO NOT CONNECT TO THIS CALL.

2 Relay Affiliation 2 IBM Security

3 Relay Affiliation Relay affiliation is used only by the automatic relay selection method. The manual relay selection method does not use relay affiliation settings. Why Setup Relay Affiliation? It helps to segment a relay deployment into separate logical groups. It limits the client endpoints to a certain set(s) of logically grouped relays. It eliminates attempts by clients in automatically trying to select undesirable relays reducing ICMP network traffic, time to selection, and selecting undesirable relays that may even be less hop counts away from desired relays. How is Relay Affiliation Implemented? Designate relays as being part of an affiliation group(s) via an AdvertisementList setting. Designate clients as being a part of an affiliation group(s) via a SeekList setting. Lists in the settings for the group names are semi-colon delimited. Naming affiliation groups? No pre-defined relay affiliation group names. You can name them whatever you wish. Avoid using special characters in the names Group names are not case sensitive, leading and trailing white space is removed. The ordering of the group names in the client's SeekList is important. The * symbol at the end of a list represent any unaffiliated computers. 3 IBM Security

4 Assigning Affiliation Group Lists Adding Advertisement Lists to BigFix Relays and the BigFix Server? Client Setting: _BESRelay_Register_Affiliation_AdvertisementList Values: Group1;Group2;* The BigFix Server should have a value of * as the BigFix Root Server accepts any and all clients outside of an affiliation group. Note: BigFix Relays and Servers do not need to be assigned a _BESClient_Register_Affiliation_SeekList. The SeekList is only used by the BigFix Client and BigFix Clients on BigFix Relays are limited to only use manual relay selection. Adding Seek Lists to BigFix Clients Client Setting: _BESClient_Register_Affiliation_SeekList Value: Group1;Group2;Group3 Note: Do not add Seek Lists to BigFix Relays. Clients on the relays register directly with the local relay. And relays should be assigned to their parent relays using manual relay selection method. 4 IBM Security

5 5 IBM Security

6 Client Behavior Uses groups whenever automatic relay selection is performed Full automatic relay selection process for each relay affiliation group in order on SeekList If it fails on all groups then it enters fail over process Failover process: Tries assigned failover relays (_BESClient_RelaySelect_FailoverRelayList) Note: If no SeekList has been assigned to the BigFix client, or if an * is added to the SeekList, the client will attempt to select relays that are unaffiliated or that have been designated in accept unaffiliated Advertisement list using the * character. 6 IBM Security

7 Relays.dat file The Relays.dat file is maintained in the actionsite on the clients Windows: \Program Files (x86)\bigfix Enterprise\BES Client\ BESData\actionsite Non-windows: /var/opt/besclient/ BESData/actionsite It stores information about the deployment's relays; it can be made readable with the Relays.dat parser tool: The actionsite needs to propagate with the relay information before a client is aware of the relays available to it and the information about the relay's affiliation groups. Example: C:\> ParseRelaysDotDat.exe Relays.dat Affiliation: Group1;* Name: relay1.bigfix.com. Port: Priority: 0 Weight: 0 Tip: Output the Relays.dat to another file Relays.txt. C:\> ParseRelaysDotDat.exe Relays.dat > Relays.txt 7 IBM Security

8 Troubleshooting Automatic Relay Selection Use ping from client to relay and relay to client to verify ICMP connectivity. Use nslookup to verify DNS resolution from client to desired relay back to client. A client may have trouble selecting a desired relay, even a close one in hops due to high network latency issues. Troubleshoot network latency issues using the trace route command. Windows: tracert / Non-windows: traceroute Google on how to troubleshoot high network latency and what to look for. Engage networking administration to assist in reducing latency issues on network segments. You may be able to work around high network latency issues by setting the following client settings to their maximum values: _BESClient_RelaySelect_MaximumTTLToPing = 50, 100, etc. (based on traceroute) _BESClient_RelaySelect_PingDelayMicroSeconds = (default: 200) After testing, if the higher values allow the client to select the desired relay, you could leave them in place, or dial them back to smaller values. If the client still cannot select desired relay with maximum values, due to high network latency, implement manual relay selection. 8 IBM Security

9 Troubleshooting Automatic Relay Selection Client logs: <client home>\ BESData\ Global\Logs Client debug logging: Windows & Non-Windows: First look for message: Date/Time: DebugMessage Beginning Relay Select First round will send ICMP pings at TTL = 1 to every relay in first affiliation group Second round will send ICMP pings at TTL = 2 to every relay in first affiliation group Third round will send ICMP pings at TTL = 3 to every relay in first affiliation group... etc: up to the maximum of TTL = 20 by default Note: ICMP pings will not be sent if the relay is in the same subnet as the client If a relay responds and is found, you will see: Received echo reply from at distance 3 The client will attempt to register to the relay and if successful you should see: Date/Time: Relay Selection: Selection Complete Date/Time: Relay Selection: Finished ( ) 9 IBM Security

10 Automatic Relay Selection Controls Task # BES Client Setting: Relay Selection Controls Minimum Retry Interval: _BESClient_RelaySelect_MinRetryIntervalSeconds (60) Maximum Retry Interval: _BESClient_RelaySelect_MaxRetryIntervalSeconds (7200) Relay Selection Interval: _BESClient_RelaySelect_IntervalSeconds (21600) Resist Failure Interval: _BESClient_RelaySelect_ResistFailureIntervalSeconds (600) Maximum TTL: _BESClient_RelaySelect_MaximumTTLToPing (20) 10 IBM Security

11 Implementing ICMP Setting Controls Autoselection Failsafe Controls: Recommendations based on number of computers in deployment and geographic distribution: 11 IBM Security

12 Implementing ICMP Setting Controls _BESClient_RelaySelect_MaximumTTLToPing _BESClient_RelaySelect_IntervalSeconds _BESClient_RelaySelect_ResistFailureIntervalSeconds _BESClient_RelaySelect_MinRetryIntervalSeconds _BESClient_RelaySelect_MaxRetryIntervalSeconds 12 IBM Security

13 Reporting Backlogs 13 IBM Security

14 Detecting Reporting Backlogs Console > BigFix Management > Deployment Health Checks Server/Relay Diagnostics page located at: Many, most, or all computers in the console have an aged last reported time and/or are grayed out. 14 IBM Security

15 Detecting Reporting Backlogs The Relay Log Windows: Program Files (x86)\bigfix Enterprise\BES Relay\logfile.txt Non-Windows: /var/log/besrelay.log Log entry in the relay log: PostResultsForwarder (HTTP Error 503): Parent relay is busy, backing off. The Client Log: Windows: C:\Program Files (x86)\bigfix Enterprise\BES Client\ BESData\ Global\Logs Non-Windows: /var/opt/besclient/ BESData/ Global/Logs Log entry in the client log: Error posting report to: ' (General transport failure. ' http failure code 503) 15 IBM Security

16 Detecting Reporting Backlogs 16 IBM Security

17 Detecting Reporting Backlogs The <BES Server - BES Relay>\FillDBData\BufferDir directory is full: Default size of BufferDir is 3MB Typically large reporting archives around 1MB in size will have taken up most of the space in the BufferDir creating a cascading failure down the relay heirarchy. These 1MB sized archived files are the result of lower level relays combining reports they received into these larger archives in order to more efficiently post them. Many 1 MB sized archived size files landing in the BufferDir may indicate a problem. 17 IBM Security

18 Reporting Backlog Cause Relay or FillDB service stopped / hung. Restart services Windows: Linux: net stop BESRelay net start BESRelay net stop FillDB net start FillDB service besrelay stop service besrelay start service filldb stop service filldb start 18 IBM Security

19 Reporting Backlog Cause Send Refresh Abuse When an operator selects 100's/1000's of computers, right clicks and chooses Send Refresh This event is logged to <BES SERVER>\server_audit.log Sun, 06 Dec :54: user "bigfixadmin" (1) sent a refresh to 5,698 computers. Console > Operators: * Set this to No for all non-master console operators in your deployment. They will be limited to sending a refresh to at most one computer at a time. * This cannot be used for master console operators. Master operators need to be trained to limit to issuing Send Refresh to at most only a few computers at a time ibm.com/support/docview.wss?uid=swg IBM Security

20 Reporting Backlog Cause Too much data, too often from a property/analysis. 1 MB sized files in the BufferDir Copy out 1 MB sized files for analysis. Use the Decompress utility: - Create a folder and put decompress.exe in the folder. - In the same folder create 2 sub folders, one called "Compressed" and one called "Decompressed" - Take the discarded files (one at a time is easiest), and put it into the folder named "Compressed" - Double-click Decompress.exe. - Open the Decompressed folder and it should contain both the compressed and decompressed version of the reports. 20 IBM Security

21 Reporting Backlog Cause Too much data, too often from a property/analysis. 1 MB sized files in the BufferDir Investigate one of the files that was decompressed. Look for properties returning large sets of data. 21 IBM Security

22 Reporting Backlog Cause Database performance bottleneck. Investigate one of the files that was decompressed. "PerformanceDataPath"[REG_SZ] Setting this key to a filename in a directory that exists will cause FillDB to write out a file containing performance data in that location. "EnabledPerformanceLogs"[REG_SZ] If this key exists and is non-empty, and the "PerformanceDataPath" the performance stream is enabled. Example: HKEY_LOCAL_MACHINE\SOFTWARE\BigFix\Enterprise Server\FillDB REG_SZ Name=PerformanceDataPath Data=c:\perflog.log 22 IBM Security

23 Reporting Backlog Cause Database performance bottleneck. FillDB insertion rates should be above 1000 rows/sec for good performance (ignore entries where only a few rows were inserted): 100 rows/sec very poor performance (something is critically wrong) 300 rows/sec poor performance (something is wrong / disk read-write problems) 1000 rows/sec OK performance 1700 rows/sec good performance 2500 rows/sec very good performance 23 IBM Security

24 Reporting Backlog Cause Database performance bottleneck. Usually due to disk performance issues. Check that read/write caching on the RAID array is set to 50/50. Check for existence of disk I/O bottlenecks or other disk performance issues. Is your disk a SAN? See Server Disk Performance ( Check to see if full transaction logging is on. Switch to simple. Run DBCC CHECKDB query for potential problems within the database. Enable SQL Server Profiler to trace expensive queries. (Tools > SQL Server Profiler) Reduce data in the database: ComputerRemover ( and AuditTrailCleaner ( tools. Check reindexing job success: SQL Server Agent > Jobs > right click on BFEnteprise Full Database Index Reorganization > View History. 24 IBM Security

25 Questions for the panel? Now is your opportunity to ask questions of our panelists. To ask a question now: Press *1 to ask a question over the phone or Type your question into the IBM Connections Cloud Meeting chat To ask a question after this presentation: You are encouraged to participate in our Forum on this topic: 25 IBM Security

26 Where do you get more information? Questions on this or other topics can be directed to the dw Answers forum: More articles you can review: BigFix Relays part 1 Open Mic: IBM developerworks wiki: oint%20manager/page/home IBM Knowledge Center BigFix welcome page: Useful link: Sign up for My Notifications Follow us: 26 IBM Security

27 Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM DOES NOT WARRANT THAT ANY SYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOUR ENTERPRISE IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY. THANK YOU Copyright IBM Corporation All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and / or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. 27 IBM Security