IBM BigFix Client Reporting: Process, Configuration, and Troubleshooting

Transcription

1 IBM BigFix Client Reporting: Process, Configuration, and Troubleshooting IBM SECURITY SUPPORT OPEN MIC To hear the WebEx audio, select an option in the Audio Connection dialog or by access the Communicate > Audio Connection menu option. To ask a question by voice, you must either Call In or have a microphone on your device. For more information, visit: 20 December 2016 NOTICE: BY PARTICIPATING IN THIS CALL, YOU GIVE YOUR IRREVOCABLE CONSENT TO IBM TO RECORD ANY STATEMENTS THAT YOU MAY MAKE DURING THE CALL, AS WELL AS TO IBM S USE OF SUCH RECORDING IN ANY AND ALL MEDIA, INCLUDING FOR VIDEO POSTINGS ON YOUTUBE. IF YOU OBJECT, PLEASE DO NOT CONNECT TO THIS CALL.

2 Panelists Presenter: Adam McDonald L2 Support Software Engineer for IBM BigFix Panelist: Nathan Hanner - L2 Support Software Engineer for IBM BigFix 2 IBM Security

3 Agenda Report Posting Process Client Reporting Symptoms of Reporting Problems Causes of Reporting Problems Isolating Reporting Problems Reporting Backlog Problems Discarded Reports 3 IBM Security

4 Report Posting Process

5 Client/Relay to Relay Report Posting Process The report posting process is mostly independent of all the other processes. This means that if reporting is not working, the following might still work: Gathering and execution of actions by the client Client and Relay downloads Gathering of sites by the Client and Relays Evaluation of content on the Client endpoints Uploading of files through the relays via the UploadManager Lack of reporting simply means that there is no longer visibility to the changes that are happening on the endpoints for the endpoints that are having a problem reporting. 5 IBM Security

6 Client/Relay to Server to Database Report Posting Process 6 IBM Security

7 Client Reporting

8 BigFix Client Reporting The client runs in a continuous evaluation loop and reports on changes to data that have been detected on the endpoint machine since the last pass of evaluation. Some of these changes include: The heartbeat Property evaluation results and values Fixlet/Task/Baseline/Site/Group/etc relevance results Action results Reports are either differential or full: Clients typically report their information in differential reports. Differential reports are reports with only the difference in additional information since the last time a specific property or relevance was evaluated. Clients sometimes need to (or are instructed to) send a full report. This full report is a full set of all of the current evaluations (and results) the client has made up to that point in time. Settings that impact Client Reporting Heartbeat interval setting Found in Console Preferences _BESClient_Report_MinimumInterval Interval in seconds below which the client will not report Default: 15 seconds Min: 0 Max: CPU usage settings: _BESClient_Resource_WorkIdle _BESClient_Resource_SleepIdle See technote : 8 IBM Security

9 The Heartbeat Interval Set in the Console Preferences File > Preferences The default is 15 minutes (Mark as offline after is recommended to be set to 3 times value of heartbeat interval). The heartbeat interval ensures that a change will occur on the client endpoint The client detects this difference causing it to have to report. This is especially necessary in deployments with little activity. The heartbeat is based off of the evaluation of a property within a built-in analysis and property within the actionsite If the client is blocked from getting to the evaluation of this analysis/property in its evaluation cycle or if the evaluation cycle is long, the heartbeat report could be delayed past the 15 minutes. 9 IBM Security

10 Symptoms of Reporting Problems

11 Symptoms of Reporting Issues Clients appear grey in the console instead of black Client properties may appear as <not reported> Actions may appear as <not reported> The Last Report time is older than the Mark offline after value in console preferences. 11 IBM Security

12 Symptoms of Reporting Issues 12 IBM Security

13 Causes of Reporting Problems

14 Causes of Broken Reporting Client: Client can t connect to deployment s network Http over port Orphaned client (unable to register with parent relay) Routing/DNS Firewalls/Proxies/other network appliances Client service off Client service hung Client machine is under resourced Client evaluating/reporting too much content Client evaluating problematic content Client taking too long to evaluate certain content The actionsite is too large Interference from 3rd party (i.e. real-time AV, HIDs) Invalid ActionSite epoch (client needs to gather current action site) Setting and configurations Reporting backlog in the deployment Relay: Relay can t connect to parent relay: Http over port Routing/DNS Firewalls/Proxies/other network appliances Relay service off Relay service hung Relay machine is under resourced Relay is overloaded with too many clients, or network connections are exhausted. Interference from 3 rd party (i.e. real-time AV, HIDs) Settings and configurations Reporting backlog in the deployment 14 IBM Security

15 Causes of Broken Reporting Server Child relays can t connect to the server: Http over port Routing/DNS Firewalls/Proxies/other network appliances BES Root Server service is off or hung FillDB service is off or hung Server machine is under resourced Server is overloaded with too many clients, or network connections are exhausted. Interference from 3rd party (i.e. real-time AV, HIDs) Settings and configurations Reporting backlog in the deployment Reports are getting rejected by FillDB Database bottleneck 15 IBM Security

16 Isolating Reporting Problems

17 Isolating the Reporting Problem 1 computer, some computers, all computers? 1 computer: Is the computer powered on and on the network (use ping and other remote network commands)? Is the BESClient service up and running? (nmap -PN remote_host, looking for port 52311) Is the BESClient service hung (check client log file activity; try restarting the BESClient service) Check client logs or client debug logging to see if the client is reporting and reporting at least as frequent as the heartbeat interval: Error posting report to: Parent relay is busy, backing off. 17 IBM Security

18 Isolating the Reporting Problem Some computers (the focus starts to shift from endpoints to local level relays): Are the computers powered on and on the network (use ping and other remote network commands)? Are their BESClient service up and running? (nmap -PN remote_host, looking for port 52311) Do the computers share a common relay or common set of relays? Check in the console. Are there too many clients registered with the common relay overloading the relay? Is the common relay s service up and running? Check the relay s logs for any errors. Check the relay s diagnostics page and watch to see if the FillDB numbers are changing, report files should be constantly arriving and leaving as they are received, compressed, and past to their parent relay. 18 IBM Security

19 Relay to Relay Posting Common Errors in relay log HTTP Error 28: Timeout was reached: Connection timed out after milliseconds No connection can be established. Error posting report to: A connection can be established but there is a problem transporting the data. Parent relay is busy, backing off. The relay s parent relay s Bufferdir is full and needs to clear before reports can be further posted Class NotASignedMessage This relay has an internal problem with deployment certificates Follow the steps in technote to fix the relay: 19 IBM Security

20 Client Evaluation Problems Reporting doesn t happen at all from a client. Or reports are not being generated and posted in a timely manner. Most likely reason, problematic content evaluation: Enable Client Debug Logging: Enable Client Usage Profiler: This article gives an analysis example. Import and Activate the Agent Performance Counters analysis: Look at the results from the Average Cycle Second and Top 10 properties to identify general problems. If the average cycle is more that 20 minutes start looking for inefficient content. It is recommended that this logging be run for a 24 hour period to give enough data to analyze the problem fully. Check the size of the actionsite BESClient\ BESData\actionsite on disk. Actionsites greater than about 25 MB might indicated too many actions by master console operators have been taken or too much custom content has been saved to the master actionsite. Reduce the number of master operator actions by stopping and deleting them 20 IBM Security

21 Client Evaluation Problems Analyze Client Debug Logging: Check the times between two Report Posted Successfully messages In this range of time, look to see what is evaluating. Most single evaluation operations happen in less than a second (typically several per second) See if there are any large gaps in time (a few minutes or more between any of the log entries), the entry on the line before the long time gap should be the culprit, for example Wed, 07 Dec :12: DebugMessage EvalLog Patching Support.45:Evaluate Property 11 Wed, 07 Dec :12: DebugMessage EvalLog Patching Support.45:Evaluate Property 12 Wed, 07 Dec :12: DebugMessage EvalLog Patching Support.344:Evaluate Property 1 Wed, 07 Dec :12: DebugMessage EvalLog Software Distribution.11:Evaluate Property 1 Wed, 07 Dec :28: DebugMessage EvalLog Software Distribution.11:Evaluate Property 2 21 IBM Security

22 Reporting Backlog Problems

23 Reporting Backlog Problem A reporting backlog in the deployment happens when one of the main top level relays is offline or is not working or when the FillDB service on the root BigFix Server cannot keep up in processing reports. If the Bufferdir is full, the downstream relay s and client s attempt to post to their parent relays and ultimately the server will all be rejected. The following message appears in the client and relay logs during this backlog condition on each attempt by the client or relay to post to its parent: "Parent relay is busy, backing off." By default, the amount of data the BufferDir directory can hold at any given time is 10,000 files with a total max size of all files 3 MB. This is configurable with the follow server settings: HKLM\Software\BigFix\Enterprise Server\PostResults "BufferDirectoryMaxSize"[DWORD]= 3 * 1024 * 1024 Defines the maximum size of the bufferdir, in bytes. PostResults will reject submissions if the bufferdir is already this large. "BufferDirectoryMaxCount"[DWORD]=10,000 Defines the maximum number of files allowed in the bufferdir. PostResults will reject submissions if the bufferdir is already this large. NOTE: Increasing the amount of data the Bufferdir can hold with this setting will not increase the speed at which reports are inserted into the database by the FillDB service. It just gives the FillDB service a larger plate to eat from. However, this larger plate may be useful when the new DatabaseBoostLevel setting is enabled IBM Security

24 Common Causes of Reporting Backlogs Send Refresh abuse: An operator selects hundreds or thousands of computers, right clicks, and clicks Send Refresh A send refresh command is sent to every computer selected instructing the computer to send up a full report containing all client property and state data Dozens of these full reports get compressed and archived together on the relays into 1 MB sized files at a time and are passed up to the root server. The megabyte sized archive files quickly fill up the Bufferdir directory (\BESServer\FillDBData\BufferDir) on the root server. Because the default size given to the Bufferdir is 3 MB you will typically see 3 of these 1 MB sized archives in the Bufferdir The database takes a longer time to process the 1MB sized archive of full reports than it does in processing reports that are typical in size (several KB) and this is what causes the backlog. A backlog could take hours to clear To confirm if send refresh is the problem check the \BESServer\server_audit.log for entries where operators have sent a refesh to a high number of computers: Sun, 06 Dec :54: user "bigfixadmin" (1) sent a refresh to 5,698 computers. Preventing Send Refresh: Train master console operators to avoid doing this Restrict non-master console operators through permissions IBM Security

25 Common Causes of Reporting Backlogs Too Much Data, Too Frequently: Copy out large reports from the FillDB Bufferdir before they are processed Or enable FillDB Carbon Copy to capture them: Then, decompress them with the Decompress Utility: In the output folder, the first file with a.0 extension is the original compressed archive, the following (starting with the 0.1 extension) are report files containing decompressed client reports Note: If you are using message level encryption (MLE), the reports need to be decrypted first in order to decompress and read them. Decryption should be configured to take place on the top level relays, not on the root server IBM Security

26 Common Causes of Reporting Backlogs Too Much Data, Too Frequently: 26 IBM Security

27 Common Causes of Reporting Backlogs Too Much Data, Too Frequently: Consider your custom properties. In the Console: Tools > Manage Properties > All Properties 1. Sort the properties by Period 2. Start considering all custom properties that have an evaluate Period of Every Report (this is the most aggressive frequency) Most bits of information from a computer endpoint do not change by the second, or even day. 3. Then, consider those that have an Evaluate period of 5 minutes, 15 minutes, 30 minutes, etc. 4. Are these custom properties/analyses returning a relatively large set of data on every evaluation? 5. Do these custom properties make requests from the OS that take a long time to completed (i.e. Long running WMI queries)? Have you tested your customer relevant in the Fixlet Debugger to see if the property to determine how much data is returned and how long it takes to return the data? 27 IBM Security

28 Common Causes of Reporting Backlogs Server/Database Bottleneck: The server may be overloaded with registered client endpoints. Disk configuration and performance: Server Disk Performance: Capacity Planning & Performance: Real Time AV interference: Add exclusions for the BigFix directories, especially the Bufferdir directory Data accumulation in the database over time: Run the Computer Remover and Audit Trail Cleaner tools about once a quarter Check the database to ensure the nightly index re-org job has been running successfully to completion. Check the general health of the database. Enable SQL Profiler or DB2 monitoring ( to trace on long running or blocking queries. Computer Remover Tool: Audit Trail Cleaner Tool: 28 IBM Security

29 Common Causes of Reporting Backlogs Server/Database Bottleneck: FillDB Performance Logging: rows/sec Very poor performance (something is very wrong) 300 rows/sec Poor performance (something is wrong or disk performance is bad) 1000 rows/sec OK performance 1700 rows/sec Good performance rows/sec Very Good performance 29 IBM Security

30 Common Causes of Reporting Backlogs Server/Database Bottleneck: The DatabaseBoostLevel setting: Use FillDB Performance Logging to capture performance of insertions for each of the tables before and after enabling the DatabaseBoostLevel setting to see the difference between the FillDB insertion rates before and after setting the DatabaseBoostLevel. 30 IBM Security

31 Discarded Reports

32 Discarded Reports The FillDB.log logs rejected reports from client(s) Discarding message from computer because it has the invalid action site epoch '04 Nov :35:18' (ought to be '12 Feb :24:54'). Typically resolves on its own over time, client needs to gather current version of actionsite. Unable to parse chunk of compressed file in buffer; discarding chunk. (Client report has no verified signer. Discarding message from computer xxxxxxxx)" EncryptedClientCAKey needs to be refreshed in the server IBM Security

33 Not Reported / Unknown

34 BigFix Client Properties Not Reported Some BigFix client properties are showing up in the console as <not reported> Client needs to be refreshed 34 IBM Security

35 Questions for the panel Now is your opportunity to ask questions of our panelists. To ask a question now: Raise your hand by clicking the hand icon in the Attendees panel on the right in the WebEx Event. The host will announce your name and unmute your line. or Type your question into Q&A panel of the WebEx Event To ask a question after this presentation: You are encouraged to participate in dw IBM Security

36 Where do you get more information? Questions on this or other topics can be directed to the product forum: Another article you can review: Technote Computers are grayed out in the console: Useful links: Get started with IBM Security Support IBM Support Portal Sign up for My Notifications Follow us: 36 IBM Security

37 THANK YOU FOLLOW US ON: securityintelligence.com xforce.ibmcloud.com Copyright IBM Corporation All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM's current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others. Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM does not warrant that any systems, products or services are immune from, or will make your enterprise immune from, the malicious or illegal conduct of any party.