QLean for IBM Security QRadar SIEM: Admin Guide QLEAN FOR IBM SECURITY QRADAR SIEM ADMIN GUIDE ScienceSoft Page 1 from 18

Transcription

1 QLEAN FOR IBM SECURITY QRADAR SIEM ADMIN GUIDE 2018 ScienceSoft Page 1 from 18

2 Table of Contents Overview... 3 QLean Installation... 4 Download QLean... 4 Install QLean... 4 Request license key... 5 High Availability license... 5 Install license key... 5 Execution Parameters... 6 Configuration... 7 Manual Execution... 7 Scheduling for Periodical Monitoring... 8 Managing QLean data... 9 Creating a backup... 9 Restoring a backup Reporting Health Markers Custom Logo Troubleshooting Appendix A: Monitoring metrics ScienceSoft Page 2 from 18

3 Overview Supported QRadar versions: QLean, previously known as Health Check Framework (HCF) for IBM Security QRadar SIEM, is a tool that allows QRadar users, administrators and security officers to perform periodical and ondemand monitoring of a range of statistical, performance and behavioral parameters of QRadar deployment including All-in- One and distributed environments QLean gathers and analyzes more than 50 different parameters (metrics) and produces an Excel report that can be delivered to one or more recipients via . This report reflects system health statistics in a tabular and graphical representation. For complete list of supported metrics please refer to Appendix A: Monitoring metrics section. NOTE: QLean is a commercial software and requires a license key to run. Free demo mode with limited functionality is also available. No license key required for running QLean in this mode. NOTE: QLean is developed by ScienceSoft Inc. and not supported by IBM ScienceSoft Page 3 from 18

4 QLean Installation QLean is distributed as a QRadar extension and can be installed in two ways: All-in-One (AiO): All QLean components run within QRadar extension container Distributed: User interface in QRadar extension and back-end on a separate Linux server NOTE: Please do not confuse QLean installation types with QRadar deployments of the same names. AiO installation is the preferable one and requires just a few steps to set up. NOTE: Please back up QLean data periodically as described in Managing QLean data section of this document. Distributed installation can be used to minimize performance impact on QRadar Console, simplify administration and avoid the risk of reporting data loss in case of possible extension container failure. If you want to create a distributed QLean installation, please contact technical support to get instructions. In order to prepare All-in-One QLean deployment please follow the steps below. Download QLean Go to Login using your IBMid Filter by Type: Application Select QLean extension Click Download button at the top right corner Save the extension zip file Install QLean Login to QRadar UI Go to Admin tab Open Extensions Management Click Add button Select Install immediately checkbox, click Browse button, locate the extension file downloaded from IBM App Exchange and click Add button Confirm on all steps and wait for installation to finish. This may take a while. Close Extensions Management window, press Ctrl+F5 to fully reload QRadar UI. New QLean tab will be added. Go to QLean tab. A message asking for QLean server deployment will be shown. Click on the Please click here link. QLean configuration page will be opened in a new window. Wait for installation to finish. Select All-in-One deployment type, enter your QRadar Console root password and click Connect button. The page will reload in a few seconds. NOTE: For more details on using IBM App Exchange and Extension Management tool, please refer to official IBM documentation: l 2018 ScienceSoft Page 4 from 18

5 Request license key In order to generate Excel reports, QLean requires a license key. You can request a commercial license by sending request to qlean@scnsoft.com. The following information required for the license key to be created: Company name Contact person Contact person position Contact Contact phone QRadar version UUID code of QRadar Console/AiO Total licensed EPS capacity In order to obtain UUID code of your QRadar Console/AiO follow the steps below: Login as root user to QRadar Console/AiO via SSH Execute command: dmidecode -s system-uuid Copy the generated alpha-numeric code to use it in your license request. If Qlean is already installed and connected to your QRadar Console, its UUID value will be shown in Configuration page: You will be sent an with a license key once you complete your purchase. High Availability license If you have a High Availability option for QRadar Console/AiO, additional license key is required to get QLean reports when the secondary instance is active. In this case, make sure to include in your request both primary and secondary UUIDs. Note that a standby instance is only accessible via SSH from an active one. Install license key In QRadar UI, navigate to QLean tab and click on the gear button at the top right corner Open QLean deployment section Click Upload license file button, locate the ZIP file received from ScienceSoft and click Open button 2018 ScienceSoft Page 5 from 18

6 Execution Parameters QLean configuration UI Enable debug Disable Data Quality analysis Disable Advanced JMX metrics Rules performance interval Offense Analysis: exclude inactive Offense Analysis: include dismissed Time range for Ariel queries Define custom FROM address for reports URL for custom logo Description Debug mode. Enhanced execution log will be created. Commands output will be written to /store/qlean/reports/hcr-yyyy-mm-dd-hh-mm-debuginfo.zip file for further review. Some information about your deployment and connected Log Sources will be extracted from configuration database. Disable Data Quality analysis in order to reduce QLean execution time. Disable Advanced Metrics in order to reduce QLean execution time. Rules performance check duration (in seconds). If not defined, will run with default 600 seconds interval (10 minutes). When set, inactive offenses will not present in Offense Analysis selection When set, Closed and Hidden offenses will be included in Offense Analysis selection Time range for Ariel queries (in hours). If not defined, will run with default 24 hours range. Define custom FROM address for reports. Define custom URL for logo in XLS report. Top Offenses Count Number of offenses to display for Top Unique Offenses metric. Default is 10. Top Assets Count Number of assets to display for Top Risky Assets metric. Default is 10. Backups number Number of backups to display for Last backups metric. Default is 5. Log Source Types Count Log Source Actions Count Rules Performance Count Number of Log Source Types to display for EPS per Log Source Type metric. Default is 10. Number of Log Sources to display for Last inactive, disabled, added, deleted, modified Log sources and Protocol Configuration Errors metrics. Default is 10. Number of rules to display for Rules Performance metrics. Default is 10. Top Reports Number of reports to display for Top Heavy Reports metric. Default is 10. Sys Notification Count Autoupdate Errors Count Number of entries to display for Last Warnings and Errors from System Notification metrics. Default is 10. Number of entries to display for Last Autoupdate Errors metric. Default is ScienceSoft Page 6 from 18

7 Configuration In order to override default parameters, open QLean configuration page through the gear button at the top right corner of QLean tab in QRadar UI. Set desired values in Execution parameters fields and click Save configuration button. All further ondemand QLean reports will be generated using this configuration, unless overridden via parameters set in QLean configuration page. To revert to out-of-the-box settings, click Reset to default button. Manual Execution Go to QLean tab in QRadar UI Click on the gear icon in the top right corner Go to Execution parameters section Define execution parameters as described in the previous section Click Run QLean button Execution status will be displayed at the top of the window. Once finished, reports list will be updated ScienceSoft Page 7 from 18

8 Scheduling for Periodical Monitoring In QRadar UI, go to QLean tab and click on the gear button at the top right corner Define required execution parameters in Execution parameters section (1) Click Add to schedule button (2) Define the schedule using drop-down lists or enter manually in the text field (3) Click Schedule button (4) Click Edit QLean tasks button to review and/or change existing crontab entries 2018 ScienceSoft Page 8 from 18

9 Managing QLean data Due to QRadar extensions architecture, an application data may be possibly lost during application update, or because of the platform s failures. Thus, taking periodic backups of your QLean reports and settings is highly recommended. The following data can be backed up and restored: XLS Reports UI reports License keys Scheduled jobs Execution parameters configuration Health Markers configuration Execution logs Debug information The following data cannot be backed up and restored via QLean Configuration interface, and therefore should be re-applied manually: Custom logo The list of recipients is configured in QRadar itself and will not be affected by any application changes or failures. Creating a backup Login to QRadar UI Navigate to QLean tab and click on the gear button at the top right corner Open QLean deployment section Click on Backup QLean data button Download the output ZIP file and save it to a safe place Restoring a backup Login to QRadar UI Navigate to QLean tab and click on the gear button at the top right corner Open QLean deployment section Click on Restore QLean data button Locate and upload the backup ZIP file 2018 ScienceSoft Page 9 from 18

10 Reporting After each run QLean can send reports via . By default, the report will be sent to the address specified under System Settings Administrative Address in QRadar Admin tab. If you want to send QLean reports via to other addresses, refer to steps below. Login to QRadar UI Navigate to QLean tab and click on the gear button at the top right corner Open QLean deployment section Click Create report recipients list button. QRadar Reference Set will be created and the button will change to Report recipients which is intended to manage addresses via QRadar standard Reference Set editor. Update Reference Set content o Press Add button o Add one or several addresses to the list NOTE: In order to temporary disable reports without removing the existing recipients, add nomail item to the Reference Set. Delete this item once you need reports again ScienceSoft Page 10 from 18

11 Health Markers QLean provides user with extended reports which contain 25 OK/Failed Health Markers in order to indicate important metrics changes in your QRadar deployment. If a marker has been triggered as Failed, a warning with description and some basic recommendations will be generated. Health Markers are being triggered on the following metrics: Deleted Log Sources: if at least one Log Source was deleted during the last days (3 days by default). Autoupdate Errors: if at least one Autoupdate failed during the last days (3 days by default). Asset Risk Level: if at least one Asset reached Risk level, which exceeds the top-10 average level on more than given threshold (70% by default). Offense Types: if at least one Offence type occurs more often (80% by default) than the top-10 average periodicity. Nightly Backups: if at least this many (0 by default) failures occurred among last 5 backups. System Notifications: if at least one error/warning was detected in System Notifications journal during the last days (3 days by default). Inactive Log Sources: if at least one Log Source became inactive during the last days (3 days by default). Disabled Log Sources: if at least one Log Source was disabled during the last days (3 days by default). Protocol Errors: if at least one Log Source has protocol configuration errors. Modified Searches: if at least one Search was modified during the last days (3 days by default). Data Integrity: if at least one event/flow data file corrupted or integrity check failed. Rules Execution Time: if at least one correlation rule executes longer than the top-10 average rules execution time on more than given threshold (70% by default) ScienceSoft Page 11 from 18

12 Rules Response Time: if at least one correlation rule responses longer than the top-10 average rules response time on more than given threshold (70% by default). Distributed EPS: if at least one managed host reached EPS utilization more than the given threshold (95% by default). Distributed FPM: if at least one managed host reached FPM utilization more than the given threshold (95% by default). Managed hosts RAM: if at least one managed host runs below the given amount (10% by default) of free RAM. Managed hosts CPU: if at least one managed host has CPU load over the given threshold (95% by default) in the last 15 minutes. Managed hosts /store partition: if at least one managed host has used /store partition space over the given threshold (90% by default). Managed Hosts Status: if at least one managed host is in state different than Active or Standby (normal operation of HA appliances). Generic DSM: if at least one SIM Generic DSM Log Source generates more than given number of events (50 by default). Unknown Events: if at least one Log Source generates more than given threshold of unknown events (90% by default). Log Source average EPS: if at least one Log Source generates more than given threshold of average Events per Second (100 EPS by default). EPS license excess: if at least one managed hosts experiences EPS license limits overrun. Default thresholds can be modified through QLean configuration page by defining required values per marker: 2018 ScienceSoft Page 12 from 18

13 Custom Logo For branding purposes, QLean allows to use a custom logo picture in report headers. The following requirements must be met: Image format: PNG Color depth: 24 bit Image size: 296x59 or less Image resolution: 72 dpi Report header background color is RGB 22, 54, 92 (HEX #16365C). For better logo readability use transparent image background or contrast colors. In order to add a custom logo follow the steps below: Prepare your logo file according to the requirements above Login to QRadar UI Navigate to QLean tab and click on the gear button at the top right corner Open Reports section Click Select logo file button and locate your logo file Click Upload button. A warning message will be shown if some requirements are not met. Click Delete custom logo button to remove your custom logo when necessary. NOTE: only one logo file can be stored at one time. Any existing logo file will be overwritten after pressing Upload button. Troubleshooting If you have problems with QLean execution or reports generation, run it with Enable debug checkbox selected. HCR-YYYY-mm-DD-HH-MM-DebugInfo.zip file will be generated and attached to the report . If processing status doesn t change for too long (e.g. stuck on some metric for several hours), click Kill button under Qlean Configuration - Execution parameters section and follow the steps below. Get the container and application logs via Download QLean Logs button under QLean Configuration - QLean deployment. Save the output qlean_debug_log.zip. Forward these files and your Excel report to the following address for investigation: qlean@scnsoft.com 2018 ScienceSoft Page 13 from 18

14 Appendix A: Monitoring metrics The following metrics are monitored with QLean: CONSOLE Console IP address Console UUID QRadar software version Version history QRadar users DEPLOYMENT: QRADAR HOSTS Managed hosts IP address HA IP address Hostname HA host role Is Console Uptime (in days) Average CPU load, % Total RAM, Mb Free RAM, % Total /store space, Gb Free /store space, % Appliance type Disk usage details DEPLOYMENT: SYSTEM HEALTH Recent backups Name Date Status Size, Mb Integrity of events (recent 24h) Number of correct files Number of failed files Number of corrupted files Integrity of flows (recent 24h) Number of correct files Number of failed files Number of corrupted files Last warnings and errors from System Notification QRadar host IP address Date Description Last auto-update errors Date Package name Description ENVIRONMENT: LOG SOURCES Number of active Log Sources Number of Log Source groups Last inactive Log Sources Log Source name Date last seen User performed the last action Last disabled Log Sources Log Source name Date last edited User performed the last action Protocol configuration errors Log Source name Date last seen User performed the last action Last added Log Sources Log Source name Date added User performed the last action Last modified Log Sources Log Source name Date modified User performed the last action Last deleted Log Sources Log Source name Date deleted User performed the last action Log Sources list Log Source name Log Source identifier Log Source activity status Last seen event date Average EPS Peak EPS Peak EPS date Protocol Log Source type Log Source extension Date added Addition type Is bulk added Status Assigned Log Source groups Description Date modified Collector 2018 ScienceSoft Page 14 from 18

15 Log Source ID ENVIRONMENT: EPS/FPM STATISTICS EPS/FPM per Managed host Managed host name Managed host IP address Average EPS for the last period (24h by default) Average FPM for the last period (24h by default) EPS license limit FPM license limit EPS utilization, % FPM utilization, % Average EPS from qradar.log Average FPM from qradar.log EPS per Log Source type Log Source type Total EPS consumption, % Average EPS Peak EPS ENVIRONMENT: RAW INBOUND EPS PER MANAGED HOST Event Collector hostname and IP address Destination Processor IP address Average EPS Peak EPS EPS license limit ENVIRONMENT: RAW INBOUND FPM PER MANAGED HOST Flow Collector hostname and IP address Destination Processor IP address Average FPM FPM license limit ENVIRONMENT: DATA QUALITY BY DEVICE TYPE Log Source type List of missing categories Category coverage, % (percentage of seen event categories against the total number of supported categories) Category name Average event severity Number of event types seen Number of event types supported Event count Event coverage, % (percentage of seen event types against the number of supported event types within particular category) ENVIRONMENT: DATA QUALITY BY LOG SOURCE Log Source type Log Source name Average event severity Number of event types seen Number of event types supported by DSM Event count Coverage, % (percentage of seen event types against the number of event types supported by particular DSM) ENVIRONMENT: UNKNOWN EVENTS AND SOURCES Unknown events Log Source IP address Log Source name Log Source type Number of received events Number of Unknown events Percentage of Unknown events SIM Generic Log DSM Source IP address Number of events received ENVIRONMENT: RUNTIME STATISTICS JMX runtime metrics Event average payload size Event average rate Event average record size Event records dropped (no connection) Event records dropped count Flow average payload size Flow average rate Flow average record size Flow records dropped (queue full) Flow records dropped count DSM Information DSM name Number of received events Events parsed, % Events normalized properly, % Events unrecognized, % Events aborted with exception, % Average event parsing time, ms ENVIRONMENT: ASSETS Number of assets Assets with the highest Risk level and Vulnerabilities count CORRELATION: OFFENSES 2018 ScienceSoft Page 15 from 18

16 Top unique offenses Offense name Number of events/flows involved Top offense closing reasons Closing reason (reason name + 80 characters of a note) Number of offenses closed by reason Offense analysis Rule name Rule description Rule tests IDs of generated offenses Offense indexes Number of entries involved (events/flows/common/offenses) CORRELATION: RULES Number of enabled rules Number of disabled rules Number of Building Blocks Number of custom rules Number of modified rules Rules performance Fired rules count Average rule execution time, sec Average rule action time, sec Average rule response time, sec CORRELATION: REPORTS The most time expensive reports Report name Estimated execution time, sec Last execution time, sec Last modified searches Search name Modification date User performed last modification SOC KPI Incident resolution time Number of offenses closed in 4h, 12h, 24h, 3d, 7d, >7d Incident response time Number of offense actions in 4h, 12h, 24h, 3d, 7d, >7d Incident closed per user Number of offenses closed by QRadar user Incident detected Number of new offenses over time Incident severity Average offense severity level over time System tuning actions Number of reference data and CRE rules modification actions over time FINE TUNING Untuned default Building Blocks Building Block name Creation date Modification date Untuned default Network Hierarchy elements Element name Element CIDR Untuned Network Hierarchy correlation rules Rule name NH elements addressed in a rule Custom DSM Unknown events Log Source name Number of unknown events Total number of received events Unknown events, % Flow sources Flow interface Number of received flows Unassigned Log Sources Percentage of Log Sources assigned to Log Source groups ADVANCED PERFORMANCE Global views performance Speed of search Search type Search name Search ID Regex relative performance Relative performance gap Custom property name Regex string User Update time 2018 ScienceSoft Page 16 from 18

17 Appendix B: Release notes New: metrics added to web UI: Untuned Building Blocks, Untuned Network Hierarchy, Top Unique Offenses, Top Heavy Reports New: Health Markers for Raw Inbound EPS and Log Source average EPS Improved: added drill-down functionality for Rules Performance in web UI Improved: XLS reports download speed Improved: EPS per Log Source Type now shows lifetime stats for entire deployment Fixed: Untuned BBs logics Fixed: drill-down for RuleWizard-related metrics Fixed: application logging to Console Fixed: inactive Add to Schedule button Fixed: XLS reports download timeout Fixed: multiple XLS, logging and exception handling issues Fixed: getting destination EP name for EC in non-debug mode Fixed: raw FPM timeline on QRadar v.7.3.0, Fixed: process hanging issue on large (over 10K Log Sources) deployments Improved: PSQL data footprint reduced Fixed: HTTP timeout increased for uploading/downloading large backup files Fixed: Minor UI issues New: added Kill button to reset hung reports Improved: more debug information via Download Qlean logs button Fixed: scheduled tasks run using proper timezone settings of QRadar Console Fixed: flow collectors selection for Raw FPM Improved: results sorting in Data Quality New: missing event categories list in Data Quality by Device Type New: drill-down functionality in Data Quality: Unknown&Stored and SIM Generic Log Sources 2018 ScienceSoft Page 17 from 18

18 1.0.0 Initial version 2018 ScienceSoft Page 18 from 18