InterConnect Let s talk about QRadar Jonathan Pechta Adam Frank 1 3/13/2017

Transcription

1 Let s talk about QRadar InterConnect 2017 Jonathan Pechta Adam Frank 1 3/13/2017

2 Agenda Appliances & Licensing Changes QRadar App Releases & SDK QRadar App Node Log Source & Flexible Licensing Tenant Improvements Reference Data & Custom Properties installation improvements HA improvements AQL improvements QRadar Network Insights Minor Updates

3 Appliances & Licensing Changes

4 New Appliances (xx48 & xx29) Two new appliance form factors have been released within the last few months. The QRadar xx29, which is the generational update of our xx28 and provides an excellent platform for all host types for our clients with medium to large scale deployments. The QRadar xx48 is the newest addition to the appliance family and is targeted at enterprise clients with very high performance requirements around data searching and analysis. The xx48 also comes with an 80K EPS certification for event collection which makes it an excellent choice for clients looking to consolidate their existing deployments or even for Telco type clients with onerous collection needs. xx48 - For large system consoles (6+ processors) and high throughput processors (80k + EPS) - 18TB of SSD, 128 Gb Ram - 28 cores xx29 - Scope: Medium to large systems - 40k EPS - 60TB of Disk, 128 Gb Ram - 20 cores

5 Appliances can be set up in any role Appliances are now role agnostic and can be purchased in bulk and set up as required by the client. Admins installing QRadar in virtual environments, 3 rd party appliances, or Cloud instances need to purchase a QRadar Software Node for each host. Appliances can be re-purposed as required No special approval is required to re-purpose hosts in different roles. Simply reinstall the software and establish the host in the role required. New appliance firmware is now available M4 appliances latest firmware version is M5 appliances latest firmware version is /13/2017

6 QRadar App Releases & SDK

7 QRadar App Developer Center What is it? The QRadar App Developer Center is a landing site for anything app development related. This is a free site and open to any user. An IBM ID is required to download the SDK from the IBM Security App Exchange. Information available: Download the SDK Learn about what s new Read documentation Blogs from developers Talk with developers on our forum App troubleshooting information Learn about submitting apps (partners only at this time) Business Partner information 7 3/13/2017

8 QRadar Assistant App The QRadar Assistant app provides recommendations, app update alerts, and QRadar Support team content. Alerts you when new app versions are available Provides app recommendations Support quick links Open Mic video links QRadar Support newsletter links Tech Tip video links List of new forum posts tagged qradar Requirements: QRadar or above Internet access required 8 3/13/2017

9 QRadar Operations App What is it? IBM QRadar Operations app pulls in information from all relevant sources within the platform to give you a single pane view into what your QRadar users are doing on the system. This enables the user to easily troubleshoot and investigate the cause of certain behaviors in the platform. Currently in Early Access. Required QRadar or above. Developed by QRadar L3 Engineering 9 3/13/2017

10 QRadar Advisor with Watson What is it? QRadar Advisor with Watson combines the cognitive capabilities of Watson and the industry leading QRadar Security Analytics Platform to uncover hidden threats and automate insights. Requirements: QRadar or above Internet access required X-Force Feeds enabled in QRadar s System Settings 10 3/13/2017

11 QRadar App Node

12 Introducing the QRadar App Node Prior to 7.3, all QRadar Apps have been installed and run on the QRadar console. This means for each QRadar App that is downloaded and installed, a small portion of the consoles resources (RAM, CPU and Storage) is used up and not available for normal console operation. With the QRadar App Node, all applications are now offloaded to a new host, relieving the console from the processing load. This also permits applications with significant resource requirements access to those resources (UBA, Spark, etc). App Nodes can reside on premise on traditional appliance type form factors or within a virtualized infrastructure. The only requirement for an App Node is that it be running RHEL 7 or CentOS 7 at the time it is added to QRadar IBM Corporation 12

13 Inter-App Communications (Named Services) In QRadar 7.3.0, we have expanded the application framework to allow applications to talk to each other. Applications can now register a service with QRadar (named) that can be accessed through the RestAPI facilities of QRadar. Once registered, any other application can simply invoke these named services to leverage the capabilities being published by the app. Service are defined with the app manifest.json and the framework registers it to the API IBM Corporation 13

14 Log Source & Flexible Licenses

15 Log Source License Limits Removed Across QRadar, we no longer enforce the number of log source limits. This applies to all QRadar versions above If you have reached the log source limit on your existing key, contact for an updated license key or for more information. New appliances will receive default keys that include an unlimited log source limit. Upgrading to QRadar will remove the previous log source license limit. 15 3/13/2017 A. QRadar Console B. QRadar Console C. QRadar Newly Installed Console

16 Flexible Licensing 7.3 contains a number of supporting changes for QRadar s new Simplified Deployment model. First, 7.3 will unlock all existing licenses within a deployment and provide the flexibility to allocate these licenses as necessary, anywhere in the deployment Further, all licensed EPS and FPM are aggregated into pools from which QRadar administrators can draw down collection and processing capacity and allocate to hosts without the need for a deploy. Finally, QRadar 7.3 has removed all aspects of log source licensing and log sources are now unlimited* in all QRadar deployments. *Up to a max value of 2,147,483,647 log sources IBM Corporation 16

17 License Pool Management Licensed EPS and FPM capacity is aggregated into a shared license pool, which can be distributed to any host in the deployment, regardless of which appliance the license is allocated to. Simplifies numerous deployment use cases such as: Deployment expansion/consolidation Deploy new or redeploy existing capacity a expansion or consolidation plans are executed Data Center migration Easily redeploy existing capacity while migrating infrastructures Service Providers MSSPs can now purchase EPS/FPM in bulk and deploy as necessary 2017 IBM Corporation 17

18 License Allocations The License allocations pane displays the current allocation of all EPS throughout the deployment. Average EPS and FPM metrics for the last 30 days are also displayed, allowing administrators to easily determine if the current allocations are adequate or if reallocation is necessary. Allocating EPS and FPM The license pool must have unallocated EPS or FPM before you can increase the allocation for a specific appliance. The EPS allocation must be a multiple of 500, and cannot exceed the maximum event rate for the appliance type. The FPM allocation must be a multiple of 10,000, and cannot exceed the maximum flow rate for the appliance type. Admin > System & License Management > Display: Licenses > License Pool Management 2017 IBM Corporation 18

19 Over-Allocations What happens if a user becomes over-allocated (expired licenses, etc)? Everything in QRadar operates as expected, data is collection, rules are run and offenses are generated. BUT Log and Flow activity views are locked out until the issue is resolved. 19 IBM Security

20 Domain Tenants Reference Data and Custom Properties

21 Reference Sets Assigned to Tenants Three options are available for new reference sets: 1. Shared available to all users. 2. Private available to the user that created the reference set. 3. Tenant available to users and administrators of that tenant. 21 IBM Security

22 Custom Properties Created by Tenants Tenants can now create their own custom properties to extract important data from event or flow payloads. Restrictions: Tenants cannot optimize custom properties and require their MSSP admin to make this change. The DSM Editor does not support tenants at this time, so all changes must be done in the older interface. 22 IBM Security

23 QRadar Installation Improvements

24 Logical Volume Manager (LVM) Support QRadar runs on a Red Hat Enterprise Linux version 7.3, which provides support for logical volume manager. LVM allows administrators to create and resize partitions or aggregate clusters of storage together to build larger volumes. For example, virtual or cloud deployed QRadar instances that find themselves in need of more and more disk space can simply add more and more space to the /store partition to meet this need. NOTE: On fresh installs, all partitions are LVM based. Upgrades DO NOT introduce LVM and a fresh install is necessary if LVM is desired/required by the user. 24 IBM Security

25 New Installation Screens for Administrators List of appliances that you can install on is provided, this means no more activation keys for administrators to manage. Administrators can install the type of appliance they want from the list. The list is filtered based on the current system s CPU and RAM specs. 25 IBM Security

26 SystemD RHEL7 introduces a new service manager called SystemD versus Init in RHEL6. systemd is not just the name of the init daemon but also refers to the entire software bundle around it, which, in addition to the systemd init daemon, includes the daemons journald, logind and networkd, and many other low-level components. How does this change apply to me? This change means that services administrators need to work with (start, restart, status, reload) use new commands. However, we have built in aliases for these commands in to QRadar In future iterations and as users get more familiar with the systemd the alias commands will be removed. Common command examples New format in QRadar service tomcat restart - systemctl restart tomcat - service hostconext status - systemctl status hostcontext - service crond stop - systemctl stop crond - service ha_manager start - systemctl start ha_manager - systemctl poweroff - systemctl reboot 26 IBM Security

27 Time Server Update In QRadar 7.2, time was synchronized across MH s and HA by using rdate, which is antiquated and required tunnelling. In QRadar 7.3, tlsdate is used on the Managed Host to access port 443 on the console to synchronize time with the console. The console in 7.3 uses chrony instead of ntpdate as the daemon to access the external time servers. The following screen shows an example of the logs from chrony:.. 27 IBM Security

28 Update Notes for QRadar Planning and preparation for upgrading: As ISO file is used to update all appliances from QRadar V7.2.8 Patch 1 ( or later) to QRadar QRadar ISO is now a hybrid ISO that supports larger sizes, which is the same as the RHEL7 from Red Hat. Minimum primary disk size is 130GB and a /store partition is required. If the primary disk size is less than 130GB or has no /store partition, you must remove host from deployment, freshly re-install it, and then re-add to the deployment. The Console appliance must be updated first, then any of the managed hosts. For HA, admins must upgrade primary first, and then secondary if host has HA. Make sure to backup any data not on /store, such as extra users because the RedHat Operating System Install wipes all system partitions. A software install upgrade requires a lot more manual steps including partitioning and requires users to follow the upgrade guide to re-install the RedHat OS before continuing on to upgrading QRadar. Interface names are different in QRadar IP is the new command (ifconfig is the old command). An example of network name that you might find on a default VMware install is ens192 Windows USB: Rufus ( as per the QRadar installation guide. Linux / Red Hat users can use the dd command to create USB keys from ISOs. 28 IBM Security

29 HA Improvements

30 High Availability QRadar V7.3.0 introduces quite a few enhancements to our integrated High Availability capabilities. Users that have architected their deployments with redundant (15xx) event collectors can now take advantage of new update framework that will reduce collection downtime to only the time it takes to fail over between and Event Collector primary and Event Collector secondary host. Prior to HA failover was essentially determined by a ping type test between the primary and secondary hosts. In 7.3.0, we have expanded our HA monitoring to include a number of other sensors or subsystems of QRadar to ensure that the host is indeed healthy and functioning correctly. Monitoring has been extended to certain key core services such as hostcontext and tomcat for example IBM Corporation 30

31 HA Enhancement Patch Enhancement for Event Collector With QRadar 7.3.0, we introduced a new form of HA on event collectors (15xx) that allow for users to now architect deployments that have reduced downtime (up to 50%) during patching operations as well as lower failover times when an actual failure occurs. Event Collector (15xx) HA pairs must be used in order to take advantage of these improvements. The replacement of DRBD by clustered filesystem allows QRadar to keep parts of the /store partition mounted on both the primary and secondary, allowing QRadar to react more quickly in the case of failure on the primary host. These improvements also provide an interesting architectural solution to reduce patching downtime in QRadar deployments. When 15xx collection pairs are used, the primary can continue to collect data while the secondary is being patched. This means that the downtime in collection is simply the failover time of the HA pair itself IBM Corporation 31

32 HA Upgrade When you apply a software fix to a High Availability pair of Event Collectors (15xx), new clustering technology is used that reduces downtime. Patch Report vm188-primary : patch test succeeded. vm188-secondary : patch test succeeded. vm188-primary : patch succeeded. vm188-secondary : patch succeeded. Event collector was offline for 68s. ===================================== =-= Patch Report for =-= vm188-secondary : patch succeeded. ===================================== Important points about doing a HA upgrade: Upgrade primary first, then start the ISO update on secondary only after the primary upgrade is finished. Admins can also update a disconnected HA secondary or fresh install host with HA recovery setup type. Off-board Storage Considerations: HA must be disconnected and removed when upgrade hosts have /store partition mounted as offboard storage (ISCSI or Fibre Channel). Upgrade detached hosts (primary first and then secondary) separately and re-add them after the upgrade. The offboard storage must be reattached during the upgrade process. 32 IBM Security

33 High Availability Sensors Sensor name Description Enabled app_services ha_services drbd_io_perf Allows an HA system to failover based on the status of application services, such as when Tomcat or hostcontext are not functioning properly or unresponsive. Allows an HA system to failover when ha services are not functioning properly, such as ha_ipaddr, ha_drbd, or ha_mount. Allows an HA system to failover when DRBD blocks IO operations on the /store partition. drbd_network_perf Allows an HA system to failover due to DRBD network latency issues. No link_status mount_status ping_status Allows an HA system to failover when a link status check with the primary HA host fails. Allows an HA system to failover when a mount point does not have the proper read/write permissions. Allows an HA system to failover when network connectivity to a ping node fails. No No No Yes No Yes 33 IBM Security

34 AQL

35 Using the SESSION BY Clause The following examples are possible uses for the SESSION BY clause: Define a user activity based on webaccess events that includes a unique combination of activities. Group events by a specific user behavior session such as website visits, downloads, or s sent. Record when users login and logout to your network, and how long they log in for. The logout closes out the related transaction that is initiated by the login. Pick an activity that you want to track and define the criteria for the session activity. The categories represent specific activities in your event logs. In this example, a new session is started for every change of user name and source IP values. For example: SELECT SessionID, starttime, username, sourceip, category FROM events WHERE username is not null SESSION BY starttime username, sourceip, BEGIN category=16001 START ' :40' STOP ' :50 SessionID start_time username sourceip category :42:03 admin :42:09 admin :42:10 admin :42:11 admin :42:27 Joe Blogs :44:11 Joe Blogs This functionality also exists as part of the QRadar API 35 IBM Security

36 Use EXPLICIT with BEGIN and END Use the EXPLICIT expression with the BEGIN and END expressions to apply more precise filtering to your transactional queries. For example, you might use the BEGIN expression with the EXPLICIT END expression to capture several (BEGIN) unsuccessful login attempts, which are followed by an (EXPLICIT END) successful login. Use the TIMEOUT and TIMEWINDOW expressions to apply time filters for the sessions in your transactional queries. Query expressions 36 IBM Security

37 Bitwise Operators Use bitwise operators for AQL to mask IP addresses and to refine the IP address search criteria. You can return all IP addresses for specific network segments or devices with specific IP addresses. You can filter your search on any or all four octets of an IP address octet. For example, you can use the bitwise AND operator to search for all IP addresses that match xxx.100.xxx.xxx to look at a specific set of IP addresses. Operators can be used on any integer fields in AQL, which allows users to do masking / manipulations of IP addresses directly. Example: Return all Source IPs that end of x.y.x.100 SELECT LONG(sourceip) AS long ip, sourceip FROM events into my_cursor WHERE (long_ip & 0x000000ff)=0x GROUP BY long_ip ORDER BY long_ip The following operators are supported: AND (&) OR ( ) XOR (#) NOT (~) LEFT SHIFT (<<) RIGHT SHIFT (>>) ZERO-FILL RIGHT SHIFT (>>>) 37 IBM Security

38 Remote Networks & Services UI

39 Remote Networks and Services The Remote Networks and Remote Services pages have been redesigned. 39 IBM Security

40 What s new for APIs

41 New Endpoints Analytics API endpoints Building blocks Custom rules Configuration API endpoints Hosts API License pool Remote networks Remote services GUI App Framework endpoints Named services Staged configuration API endpoints License pool Remote networks Remote services Services endpoints DNS lookups DIG lookups WHOIS lookups 41 IBM Security

42 QRadar Network Insights

43 QRadar Network Insights Filling in the important gaps Basic Enriched Flows Content Enriched Flows What is out there? Who is talking to whom? Source and destination Info Network protocol Byte / packet counts Time of first / last packet Quality of service Virtual LAN information Application identification Usernames, s, chat IDs URLs Search arguments Host information HTTP analysis DNS queries / responses File name, type, size, hash Personal information detection Confidential data detection Embedded scripts Redirects Configurable content-based suspect content What files and data are being exchanged? Do they look malicious? Do they contain any important or sensitive data? Is this new threat on my network? Web categories IP reputation Configurable suspect content If so, it where is it and what did it do? IBM QRadar Network Insights (QNI) provides in-depth visibility into network communications on a real-time basis to extend the capabilities of your QRadar deployment. Logs and traditional network flow data are not providing enough visibility. Records application activities, captures artifacts, and identifies assets, applications and users participating in network communications Configurable analysis from network traffic for real time threat detection and long-term retrospective analysis New Appliance with out-of-the-box content on the App Exchange for fast time to value and best practices 43 IBM Security

44 QRadar Network Insights - Content Types Unparalleled real time visibility covering blind spots and complexities in log data, which reveals previously hidden threats, and malicious behaviors Enables far greater and easier attack visibility from malware infiltration, lateral movement and data exfiltration within an organization Scalable to meet any analytics demands and 10G network speeds Tightly integrated with QRadar Incident Forensics for post incident investigations and threat hunting activities Seamless integration across the QRadar platform: Extends QRadar flow capabilities QNI analysis fuels QRadar capabilities, content and Apps Derives sense events for User Behavior Analytics for improved insider risk assessments 44 3/13/2017 Content Pack Available Custom AQL Function 1 Custom Property 25 Custom QIDMap Entry 4 Custom Rule 8 Reference Data Collection 4 Report 3 Saved Search 4 QRadar Network Insights helps to identify: Insider Threats Malware Detection & Analysis Discover the network and services Phishing & Campaign Detection Data Exfiltration Detection Identify Compliance Gaps

45 Minor changes

46 Deployment Editor is Removed The Deployment Editor is removed in QRadar V IBM Security

47 47 3/13/2017 InterConnect 2017

48 Notices and disclaimers Copyright 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed as is without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply. the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and 48 3/13/2017

49 Notices and disclaimers continued Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera, Bluemix, Blueworks Live, CICS, Clearcase, Cognos, DOORS, Emptoris, Enterprise Document Management System, FASP, FileNet, Global Business Services, Global Technology Services, IBM ExperienceOne, IBM SmartCloud, IBM Social Business, Information on Demand, ILOG, Maximo, MQIntegrator, MQSeries, Netcool, OMEGAMON, OpenPower, PureAnalytics, PureApplication, purecluster, PureCoverage, PureData, PureExperience, PureFlex, purequery, purescale, PureSystems, QRadar, Rational, Rhapsody, Smarter Commerce, SoDA, SPSS, Sterling Commerce, StoredIQ, Tealeaf, Tivoli Trusteer, Unica, urban{code}, Watson, WebSphere, Worklight, X-Force and System z Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: /13/2017