Open Mic #13: Log Source Protocols

Transcription

1 IBM Security QRadar April 28, 2016 Open Mic #13: Log Source Protocols Panelists Colin Hay QRadar Ecosystem Team Lead Chris Collins Integration Team Lead L3/Maintenance Randika Upathilake Integration Team Lead/Roadmap Adam Frank Principal Solutions Architect Jeff Rusk Development Manager, QRadar L3 Engineering Jamie Wheaton QRadar Integration WinCollect Team Lead Joey Maher Support Technical Lead Jonathan Pechta Support Technical Writer Reminder: You must dial-in to the phone conference to listen to the panelists. The web cast does not include audio. USA toll-free: Participant passcode: Slides and additional dial in numbers: NOTICE: By participating in this call, you give your irrevocable consent to IBM to record any statements that you may make during the call, as well as to IBM's use of such recording in any and all media, including for video postings on YouTube. If you object, please do not connect to this call.

2 Introduction During this presentation, we discuss some of the basics on how QRadar uses log source protocols to collect event data. Each protocol in QRadar is unique, whether it be configuration properties, error messages, and use cases for data collection. Participants: If you want us to stop to ask a question or discuss a topic further, make sure you ask in the webcast chat. 2

3 How many log source protocols are in QRadar? There are currently ~52 log source protocols in QRadar that are available to administrators. Some of these are used for general purpose collection and some are product / vendor specific. As of 7.2.6, we support 315 log source types out of the box, with 174 of them supporting auto detection via traffic analysis. There are basically three categories of protocols and these will be the focus of today s discussions. Listening Protocols Syslog TLS Syslog TCP Multiline Syslog UDP Multiline Syslog Syslog Redirect Polling Protocols JDBC Log File SMB Tail Specialty Protocols (other) REST API Protocols Amazon AWS Microsoft Office 365 Subscription Protocols (SDEE) 3

4 Protocols available in QRadar Ahnlab Policy Center JDBC Amazon AWS REST API Ariel REST API Blue Coat WSS REST API Cisco NSEL EMC VMWare HTTP Receiver IBM Fiberlink REST API IBM Security Identity Manager JDBC IBM SmartCloud Orchestrator REST API IBM Tivoli Endpoint Manager SOAP JDBC Sophos Enterprise Console JDBC Juniper Security Binary Log Collector Juniper NSM OPSEC/LEA Log File MQ JMS Netskope Active REST API ObserveIT JDBC Office 365 REST API Okta REST API Oracle Database Listener Salesforce REST API SDEE Seculert Protection REST API JDBC - SiteProtector SMB Tail SNMPv1 SNMPv2 SNMPv3 Sourcefire Defense Center Estreamer Syslog Syslog Redirect TCP Multiline Syslog TLS Syslog UDP Multiline Syslog VMWare vcloud Director WinCollect WinCollect File Forwarder WinCollect Juniper SBR WinCollect Microsoft DHCP WinCollect Microsoft ISA/NPS WinCollect Microsoft ISA/Forefront TMG WinCollect Microsoft SQL WinCollect NetApp Data ONTAP Microsoft DHCP Microsoft Security Event Log Microsoft Security Event Log Custom Microsoft Security Event Log over MSRPC Microsoft Exchange Microsoft IIS Forwarded *Protocol Common 4

5 Events FAQ and terminology

6 Events FAQ What is an Event? In QRadar terms, an event is a message we receive and process from a device on your network, that represents the log of some particular action on that device. An example would be an "ssh login" on a Linux server, a VPN connection to a VPN device, or a firewall deny logged by your perimeter firewall. These actions represent something that occurs at an instance of time and are logged. What is a log source protocol? A log source protocol is the framework for getting event data off the wire. This can include listening for data or complying with specifications to establish connections, managing input queues, and getting the data to the event pipeline (ECS). 6

7 Events FAQ (continued) What is a DSM? A DSM is a Device Support Module. DSMs are created for security products, software, or appliances to allow QRadar to understand (parse) the messages (event format) provided by the device and categorize the data properly in the user interface. What is Traffic Analysis (TA)? Traffic Analysis, also known as Auto Discovery or Auto Detection, allows QRadar to detect and create new log sources based on incoming data streams. Traffic Analysis works with the Syslog protocol (both TCP and UDP) as well as SNMP protocol messages. When QRadar starts receiving data from an new address or even from an address we have already create data for, but another version of data, it sends that data over to the "traffic analysis engine" for processing. If the data is not something we recognize, an unsupported device, or if the data is sent at a very low rate (4-5 events per minute), we will likely fail auto discovery. Events from that device/system will be assigned to the SIM Generic-2 Log Source until the administrator resolves the issue or creates a log source manually. 7

8 How is the source IP or destination IP address determined? When QRadar receives and processes event data, it must assign an IP address to the Source IP and Destination IP fields. QRadar looks in the following locations, to determine the IP address to use, in the following order: 1. IP address fields in Payload Information The availability of more detailed IP address information depends on each Log Source Type, as well as the events themselves, as not all events will contain IP address fields. If the source IP address in available, the Source IP field will be updated with this information. If source IP information is not available, then it will remain as it was last set in the previous step. The same is true of destination IP information. If destination information is not available then it will remain set as either the Syslog hostname field, if an IP was available, else it will remain set as the source of the packet. 2. The hostname field in the Syslog header QRadar will look for an IP address in the hostname field of the Syslog header, if available. Note: Not all Syslog sources use proper headers. If an IP address is found, the Source IP and Destination IP fields are updated with this IP address. If the hostname field contains a textual hostname, then it is not used. QRadar will not do a DNS lookup on a hostname, as it would take too much time to do for every event, and would affect pipeline throughput capacity. 3. The source IP address of the packet the event came from, when received by QRadar The Source IP and Destination IP fields are set to the source IP address of the packet itself. This would be the device that sent the data to QRadar. If you are using an existing, centralized Syslog server to forward events to QRadar, you may often see the IP address of the Syslog server in the Source IP and Destination IP fields. The best ways to avoid this is to do one of the following: Set the Log Source device to send Syslog directly to QRadar. Preserve the initial Syslog headers, and have the originating devices configured to send an IP address in the hostname filed of the Syslog header. Reconfigure your Syslog server to prepend a new Syslog header to the events it forwards to QRadar, with the originating devices IP address in the hostname header field. 8

9 Traffic Analysis in action! Detecting new log source and adding it to the traffic analysis engine. [ecs-ec] [[type=com.eventgnosis.system.threadedeventprocessor][parent=csd.ibm.lab:ecsec/ec/trafficanalysis1/trafficanalysis]] com.q1labs.semsources.filters.trafficanalysis.trafficanalysisfilter: [INFO] [NOT: ][ /- -] [-/- -]Now collecting traffic analysis statistics for address paloalto.paseries.test.com Attempting to create new log source. [ecs-ec] [[type=com.eventgnosis.system.threadedeventprocessor][parent=csd.ibm.lab:ecsec/ec/trafficanalysis1/trafficanalysis]] com.q1labs.semsources.filters.trafficanalysis.trafficanalysisfilter: [INFO] [NOT: ][ /- -] [-/- -]Attempting to create a new sensor device: Log Source Type = Contivityv2, Address = paloalto.paseries.test.com... Pausing TA on that log source identifier (syslog host header) [ecs-ec] [[type=com.eventgnosis.system.threadedeventprocessor][parent=csd.ibmlab:ecsec/ec/trafficanalysis1/trafficanalysis]] com.q1labs.semsources.filters.trafficanalysis.trafficanalysisfilter: [INFO] [NOT: ][ /- -] [-/- -]Pausing traffic analysis on address paloalto.paseries.test.com while waiting for another device to be created... If a log source fails to auto discover / auto detect [ecs-ec] [[type=com.eventgnosis.system.threadedeventprocessor][parent=csd.ibm.lab:ecsec/ec/trafficanalysis1/trafficanalysis]] com.q1labs.semsources.filters.trafficanalysis.trafficanalysisfilter: [WARN] [NOT: ][ /- -] [-/- -]Unable to determine associated log source for IP address < >. Unable to automatically detect the associated log source for IP address. 9

10 Listening Protocols (Syslog)

11 Standard fields in the log source user interface Log sources that do not auto discover in QRadar must be manually created by the administrator. These are the common fields for a Syslog protocol log source. Remember that a single product might support multiple protocols. A common example is Syslog and TLS Syslog. Even though QRadar might list two protocol options, it is up to the administrator to ensure that the product is actually capable of providing data using the selected protocol. 11

12 Standard fields in the log source user interface (continued) Log Source Name Log Source Description Log Source Type Protocol Configuration** Log Source Identifier** Enabled Credibility Target Event Collector** Coalescing Events** Incoming Payload Encoding Store Event Payload 12

13 The Syslog Protocol QRadar s Syslog port is TCP or UDP 514. For QRadar appliances that collect or process events, TCP/UDP port 514 (ecs-ec) listens on all interfaces. Ports are enabled when appliances are added to the deployment. QRadar supports Syslog RFC3164 and RFC5424. Termination characters in a Syslog payload will truncate the event. QRadar will then create a secondary event under SIM-Generic as Stored from the remaining event payload data. UDP Syslog truncates payloads at 1024 bytes by default. TCP Syslog truncates payloads at 4096 bytes by default. This value is configurable in the QRadar System Settings. 13

14 TLS Syslog Protocol TLS Syslog is a secure protocol that encrypts payload data for transport. TLS requires certificates, which can either be provided to QRadar or a default generated certificate can be used. Private keys for provided certificate path must be a DER-encoded PKCS8 key. The configuration fails with any other key format. One TLS log source can support up to 1,000 auto discovered TLS log sources on a single appliance is the default port per specification, however, alternate ports are supported. A Deploy Full Configuration is required if an saved/existing port is changed. To verify if the TLS port is listening, type: netstat -tulpn grep

15 TCP Multiline Syslog The TCP Multiline Syslog protocol uses regular expressions to define the start, end, or start and end of a TCP Syslog payload that spans multiple lines to create a single-line event for QRadar. If you only specify an Event Start Pattern as your regular expression, the event will be created using the first occurrence of the start pattern and then the next match that occurs from the regex pattern you entered. Everything in between is created as a single-line event. This same logic applies to only having an Event End Pattern defined in the log source configuration. Depending on your event payload, you might not require both a start and end event pattern. The Event Formatter is only used for specific scenarios, such as TCP Syslog event payloads from a system sending Windows Multiline data. 15

16 UDP Multiline Syslog The UDP Multiline Syslog protocol uses regular expressions to define a repeating value from the UDP Syslog payload to logically create a single-line event for QRadar. <134>May 14 19:04:49 DB1 [25646]: LOG: data_request user: testadmin DB: PCI-2 <134>May 14 19:04:49 DB1 [25646]: select o.id,o.type,o.serial_number,o.card_number,o.currency_id,to_char,o.sum_in NOTE: The flat file version of this same functionality can be found in the Log File Protocol using the Event Generator: ID-Linked Multiline. The Event Formatter is only used for specific appliances, such as Cisco ACS Multiline event data. 16

17 Syslog Redirect The purpose of the Syslog Redirect is to deal with log sources that either do not correctly set a syslog header, providing the wrong log source identifier in the header, or a log source identifier that messes up our internal message routing (ie ). Syslog Redirect takes the value from the Log Source Identifier RegEx and the matched value (if anything) to substitute that value in as the log source identifier internally (as the packet IP). If the event has a valid syslog header we are overriding it with a new one that contains the new origin address. If an event does not have a match to the RegEx it should be posted as a regular syslog event and passed to ECS in its unmodified state. Syslog Redirect matched event = <Redirect header with injected value><default Syslog header><payload for event data> Good for correcting systems that obscure the true event source IP: - Load balancers - SymantecServer - Syslog forwarders / Syslog relay - Virtual systems - Log aggregators / 3 rd party hosts 17

18 Polling Protocols (JDBC / Log File)

19 JDBC Protocol The JDBC protocol can retrieve data from MSDE, Postgres, Oracle, Sybase, DB2, or Informix databases. The protocol can query true databases or views created by administrators specifically for QRadar polling. Requires an incrementing Compare Field value to work properly. This can be an ID, timestamp, or any increasing numeric value that is not reset. The log source identifier must be unique. You cannot have two log sources with However, you can specify a table to make the value unique: table databasename@hostnameorip 19

20 JDBC Protocol (continued) Predefined queries can be used to collect data from specific databases. Instead of requiring the admin to create a view, the predefined query can send complex commands to retrieve data easily. Select list is the data being retrieved during the poll of the database. EPS throttle is set by default to 20,000 Events per second. This value can exceed your license, but very large initial polls can cause performance issues. If your database requires or forces an encryption check to connect, the Use SSL check box is required. 20

21 Log File Protocol The Log File protocol is a file retrieval system to use FTP, SFTP, SCP, or AWS to connect to a remote log repository or appliance and copy the files to /store/tmp in QRadar for processing. The Log File protocol is not intended for files that append data, but instead for files that rollover or are created new on an interval, such as a timestamp. Ignore Previously Processed File(s) is used to track each file that has been imported to ensure that the file is not processed a second time, which would create duplicate files. If you uncheck this box, the Log File protocol will continue to import and process all files that match the File Pattern / regular expression. 21

22 Log File Protocol (continued) Event Generators allow QRadar to handle special formatting scenarios that might be in the log file or logs that need special handling or processing before they can be sent to be parsed in ECS. If Run on Save is checked, the Log File attempts to retrieve data immediately after the log source is saved. Tip: When you see the term Pattern in QRadar this almost always indicates a regular expression pattern is required and not a glob, such as * (wildcard) value. Recurrence defines the frequency with the protocol attempts to retrieve data. The shortest possible interval is 15M. 22

23 Log File Protocol (continued) Remote Directory defines the file location of the logs. If the path for the remote directory is left blank, the protocol starts from the user s home directory. You logs must reside in this directory or a subdirectory to be successful if you leave the remote directory blank. Recursive allows the protocol to traverse subfolders of the remote directory to look for files that match the file pattern (regex). Processors are used to extract binary files, such as zip, tar, tgz, etc before writing the file to /store/tmp on the QRadar appliance. Folder Separator is used when the remote file system uses non-standard separators. The Log File protocol can use the correct separator to navigate the remote file system. 23

24 SMB Tail Protocol The SMB Tail protocol allows administrators to receive new data appended to a file on a remote Samba share. The protocol keeps track of the file size or data modified time for any files that match the File Pattern (regular expression). The SMB Tail protocol is the base for a number of other QRadar protocols, such as Windows IIS protocol, Microsoft DHCP protocol, Microsoft Exchange protocol, and more. There are many use cases for SMB Tail. In general we utilize SMB Tail for Windows applications that do not have their own method for sending their events, such as Syslog. The Windows operating system also needs to have a 'share' configured to ensure that Samba can connect and read the files. For example, we use SMB Tail for things like Microsoft Exchange, Microsoft DHCP, Microsoft IIS, etc. TIP: To run the smbclient tool, type /usr/bin/smbclient -L host Where host is the server you are attempting to connect to. Alternately, you can use /user/bin/smbclient \\\\netbios_name\\c$ -U domain\user 24

25 Tips and Performance Suggestions

26 Tips and performance suggestions 1. When using the Log File protocol, query in large directory structures can take longer than expected when you attempt to recursively search. The protocol will attempt to traverse the entire directory structure to locate files. Having your event log files in a dedicated directory structure is never a bad idea. 2. Disabling functional log sources for protocols like JDBC can unintentionally cause license spikes and performance. If an administrator configures a polling log source, like JDBC or Log File protocol, when re-enbabled these protocols can pull in massive amounts of data. So, be wary of disabling log sources long term. 3. Creating a unique QRadar user is usually a good idea. This allows you to audit the data that QRadar log source protocols are retrieving by providing a unique user name. 4. Remember that if you install a protocol manually, the instructions in the commandline inform administrators to: 1. Complete a Deploy Full Configuration, then 2. Restart Tomcat (Admin tab > Advanced > Web Server Restart). 26

27 Tips and performance suggestions (continued) 5. When you have multiple log sources defined in parsing order, it will try the first one, and if it doesn't match, tries the next. If it does not match the second or next value, it will then store that event to the Primary log source as a stored event.. If it does match the second, it will then store properly as the second. 6. Some event sources do not allow admins to configure Syslog ports below TCP 1024, for example like Cisco ASA. If you have a scenario like this, you can use iptables in QRadar to preroute and redirect data from a non-standard port to QRadar s supported Syslog port, which is TCP / UDP 514. However, you cannot change QRadar s default Syslog port. 7. Log sources can be bulk added, but only using select protocols such as Syslog and Windows protocols (WinCollect, MSRPC, and WMI). 8. If you have older log sources, it can be better to disable them than to delete them. The reason behind this is that in both circumstances the data is still on the QRadar appliance. However, if the log source is deleted, you cannot quick filter by log source group or log source name as those references no longer exist. If you think you might want to look up older data, think about disabling older log sources. Also, deleted log sources can auto discover again if they continue to send event data. 27

28 Specialty Protocols (APIs)

29 Specialty Protocols Amazon AWS CloudTrail In the Amazon AWS log source configuration, the Access Key ID is equivalent to the Public Key field. The Secret Access Key as defined in Amazon's AWS documentation should be configured to use the Access Key field in the log source configuration. Copy the.der certificate to the opt/qradar/conf/trusted_certificates directory of the QRadar appliance that manages the Amazon AWS CloudTrail log source. 29

30 Specialty Protocols What do you want to talk about? 30

31 Questions & discussion

32 Advanced questions from the forums Q1: I know we can use the Microsoft Security Event Log over MSRPC to pull events that are written to the default Windows Logs (e.g. Application, Security, Setup, System...). Will there be any way to pull events that have been configured to write under the Applications and Services Logs using a preloaded protocol? I don't know of any way besides configuring a WinCollect agent to manually pull these logs. This would be a nice to have since there are instances where we configure logs to be written to a separate location to keep them separate from all the other "noise". Q2: What is the best solution to retrieve logs from file located on a Windows shared folder? Q3: When using Log File protocol to retrieve a log from a Unix file: If I check "Ignore Previously Processed File(s)" I will loose then some logs from the latest open file where new logs are written. If "Ignore Previously Processed File(s)" is not checked, the log source will always retrieve the same events and logs will be duplicated. 32

33 Questions for the panel? Now is your opportunity to ask questions of our panelists. To ask a question now: Press *1 to ask a question over the phone or Type your question into the SmartCloud Meetings chat To ask a question after this presentation: You can ask questions in our General forum: 978a-403e-83ae-d4edbebab3ec&ps=25 33

34 Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access. IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM DOES NOT WARRANT THAT ANY SYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOUR ENTERPRISE IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY. THANK YOU Copyright IBM Corporation All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and / or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.