McAfee SIEM HA Receivers

Transcription

1 Product Guide Revision A McAfee SIEM HA Receivers

2 COPYRIGHT Copyright 2015 McAfee, Inc., 2821 Mission College Boulevard, Santa Clara, CA 95054, , TRADEMARK ATTRIBUTIONS Intel and the Intel logo are registered trademarks of the Intel Corporation in the US and/or other countries. McAfee and the McAfee logo, McAfee Active Protection, McAfee DeepSAFE, epolicy Orchestrator, McAfee epo, McAfee EMM, McAfee Evader, Foundscore, Foundstone, Global Threat Intelligence, McAfee LiveSafe, Policy Lab, McAfee QuickClean, Safe Eyes, McAfee SECURE, McAfee Shredder, SiteAdvisor, McAfee Stinger, McAfee TechMaster, McAfee Total Protection, TrustedSource, VirusScan are registered trademarks or trademarks of McAfee, Inc. or its subsidiaries in the US and other countries. Other marks and brands may be claimed as the property of others. LICENSE INFORMATION License Agreement NOTICE TO ALL USERS: CAREFULLY READ THE APPROPRIATE LEGAL AGREEMENT CORRESPONDING TO THE LICENSE YOU PURCHASED, WHICH SETS FORTH THE GENERAL TERMS AND CONDITIONS FOR THE USE OF THE LICENSED SOFTWARE. IF YOU DO NOT KNOW WHICH TYPE OF LICENSE YOU HAVE ACQUIRED, PLEASE CONSULT THE SALES AND OTHER RELATED LICENSE GRANT OR PURCHASE ORDER DOCUMENTS THAT ACCOMPANY YOUR SOFTWARE PACKAGING OR THAT YOU HAVE RECEIVED SEPARATELY AS PART OF THE PURCHASE (AS A BOOKLET, A FILE ON THE PRODUCT CD, OR A FILE AVAILABLE ON THE WEBSITE FROM WHICH YOU DOWNLOADED THE SOFTWARE PACKAGE). IF YOU DO NOT AGREE TO ALL OF THE TERMS SET FORTH IN THE AGREEMENT, DO NOT INSTALL THE SOFTWARE. IF APPLICABLE, YOU MAY RETURN THE PRODUCT TO MCAFEE OR THE PLACE OF PURCHASE FOR A FULL REFUND. 2 McAfee SIEM HA Receivers Product Guide

3 Contents Preface 5 About this guide... 5 Find product documentation Introduction 7 SIEM High Availability Receivers... 7 High Availability Architecture... 8 Workflow High Availability Components 11 Corosync Pacemaker IPMI Configuration Files High Availability Setup 21 Requirements Cabling the Receivers Configuration Steps High Availability Tools 31 The User Interface The Command Line Interface Troubleshooting 34 Causes of Failover Using the configuration files to locate the issue Scenarios that would require a manual return to an online status References 55 McAfee SIEM HA Receivers Product Guide 3

4 4 McAfee SIEM HA Receivers Product Guide

5 Preface This guide provides the information you need to configure, use, and maintain your McAfee product. About this guide This information describes the guide's target audience, the typographical conventions and icons used in this guide, and how the guide is organized. Audience McAfee documentation is carefully researched and written for the target audience. The information in this guide is intended primarily for: Administrators People who implement and enforce the company's security program. Users People who are responsible for configuring the product options on their systems, or for updating their systems. Reviewers People who evaluate the product. Conventions This guide uses the following typographical conventions and icons. Book title or Emphasis Bold User input, Path, or Code Hypertext Note: Tip: Important/Caution: Warning/Danger: Title of a book, chapter, or topic; introduction of a new term; emphasis. Text that is strongly emphasized. Commands and other text that the user types; the path of a folder or program; a code sample. A live link to a topic or to a website. Additional information, like an alternate method of accessing an option. Suggestions and recommendations. Valuable advice to protect your computer system, software installation, network, business, or data. Critical advice to prevent bodily harm when using a hardware product. McAfee SIEM HA Receivers Product Guide 5

6 Introduction Find product documentation Find product documentation After a product is released, information about the product is entered into the McAfee online Knowledge Center. Task 1 Go to the Knowledge Center tab of the McAfee ServicePortal at 2 In the Knowledge Base pane, click a content source: Product Documentation to find user documentation Technical Articles to find KnowledgeBase articles 3 Select Do not clear my filters. 4 Enter a product, select a version, then click Search to display a list of documents. 6 McAfee SIEM HA Receivers Product Guide

7 1 Introduction The SIEM architecture allows for the implementation of a High Availability infrastructure at the Receiver level to ensure continuity of data collection in the event of a critical failure. Acting in a Primary and Secondary role, the Receivers can switch these roles should the current Primary fail under certain conditions. Contents SIEM High Availability Receivers High Availability Architecture Workflow SIEM High Availability Receivers High Availability is provided by two Receivers acting in a Primary and Secondary role for data collection, with the designated Primary fulfilling the purpose of being the active Receiver. The collection of data is processed in the same manner as if it were a standalone Reciever, with the exception of the physical network interface configuration. Benefits of the High Availability architecture By incorporating a High Availability architecture as part of the SIEM estate, the following benefits can be derived; Automatic failover in the event of a critical failure on the Receivers. Minimal downtime in the event of a hardware failure requiring parts or product replacement. Guaranteed continuity of data collection in the event of a critical failure on the Receivers. The Primary Receiver The Receiver that is designated as Primary is responsible for the collection of data for those data sources configured on the HA pair. It performs this role in the same manner as that of a single, standalone Receiver. The Secondary Receiver The Receiver that is designated as Secondary continuously monitors the Primary for any critical service or hardware failure that might render it inoperative and unable to perform it s role in the collection of data. At the point that such a failure is detected by the Secondary, it will fence off the Primary receiver and assume the Primary role. McAfee SIEM HA Receivers Product Guide 7

8 Introduction High Availability Architecture High Availability Architecture The operation of the High availability between the two Receivers requires the use of five separate physical interfaces as illustrated in the following diagram: Diagram 1: NIC Connectivity Diagram Legend Item ESM Data Source NIC MGMT NIC IPMI Card IPMI NIC HB NIC Explanation Enterprise Security Manager The interface used for collection of data from the Data Sources The interface used for communication with the ESM and other devices. It can also be used for remote SSH access to the Receiver. The dedicated IPMI interface used by the IPMI subsystem. The network interface used by each Receiver to communicate directly with the IPMI dedicated interface. The network interface used for the heartbeat that monitors the status of the other Receiver. 8 McAfee SIEM HA Receivers Product Guide

9 Introduction High Availability Architecture Network Interface Overview Each of the network interfaces as shown in the diagram performs a specific role within the HA architecture. The combination of each of these allows the High Availability to function. The Management Interface The management interface is responsible for the communication between the Enterprise Security Manager and any other attached devices (including any configured external storage) on the Receivers. The Data Source Interface The Data Source interface is only active on the current Primary configured Receiver. It consists of a single IP and MAC address that is shared between both Primary and Secondary. While the Primary is up and active, it holds this shared IP and MAC address and is responsible for the data collection. In the event of a failure on the system that results in a fail over, so that the Secondary now becomes Primary, the shared IP and MAC address become active on the new Primary device and is removed from the failed Receiver. The IPMI Card Interface IPMI is an acronym for Intelligent Platform Management Interface and uses a dedicated NIC on the Receiver motherboard with BIOS integration. It is ultimately responsible within the High Availability configuration for fencing a failed Receiver and ensuring that only one is currently acting as Primary at any one time. The IPMI NIC Interface The IPMI NIC interface is used on each Receiver to allow the other processes involved in providing High Availability to talk directly to the IPMI sub system. This interface is a directly cabled connection with the IPMI card interface. The Heartbeat Interface This comprises of a directly cabled connection between both designated heartbeat interfaces on the Receivers. There is constant communication between the two Receivers over this connection and in the event that communication stops from either one the other Receiver will recognise this as a failure and the High Availability failure process is initiated. McAfee SIEM HA Receivers Product Guide 9

10 Introduction Workflow Workflow During normal operations, the communications and data exchange takes place as follows: Diagram 2: Workflow The ESM sends any configuration changes and policy to both Receivers. It periodically collects events from the currently designated Primary Receiver and collects status updates from both Receivers in the HA pair. Each Receiver monitors and reports its own health status. The Primary and Secondary communicate constantly over the heartbeat and management interfaces. Any certificates received as required for certain Data Source configurations and operation (OPSEC, Estreamer) are shared with the other Receiver in the in the HA pair. Data Source communication takes place on the currently active Primary Receiver on the Data Source interface. 10 McAfee SIEM HA Receivers Product Guide

11 2 High Availability Components To implement the High Availability capability in the Receivers a number of applications are used that provide this functionality. Contents Corosync Pacemaker IPMI Configuration Files Corosync Corosync is described as a Group Communication System with additional features for implementing high availability within applications. (Clusterlabs, n.d.). Corosync acts as the communication layer for the other applications used for High Availability to take place. At the core of Corosync is the Totem Single Ring Ordering and Messaging Protocol. This has the benefits of ensuring the following: Messages are transmitted and received in correct order and reliably. This is important because it is of no use receiving notification of a service or member being down after it has recovered. It establishes cluster membership, so in the case of the Receiver HA it identifies both configured Receivers as being part of this particular cluster. It establishes quorum, which ensures that both Receivers have joined the cluster and are operational. If we look at the corosync.conf on one of the Receivers, it will be along the lines of the following: totem { version: 2 secauth: on threads: 0 rrp_mode: passive transport: udpu rrp_token_expired_timeout: 75 McAfee SIEM HA Receivers Product Guide 11

12 High Availability Components Corosync interface { ringnumber: 0 bindnetaddr: mcastport: 5405 member { memberaddr: } member { memberaddr: } } interface { ringnumber: 1 bindnetaddr: mcastport: 5405 member { memberaddr: } member { memberaddr: } } } logging { fileline: off to_stderr: no to_logfile: yes to_syslog: no logfile: /var/log/corosync.log debug: off timestamp: on } 12 McAfee SIEM HA Receivers Product Guide

13 High Availability Components Pacemaker aisexec { user: root group: root } service { # Load Pacemaker name: pacemaker version: 0 # use_logd: yes } The two main directives to consider in the above configuration are the totem and service directives. The totem directive is used to pass arguments to the totem protocol. In our Receivers we have a level of redundancy as two rings have been specified, each identified by a separate and consecutive ring number. Ringnumber 0 is for the heartbeat connection and ringnumber 1 contains the IP addresses of the configured management ports of both Receivers. The service directive instructs Corosync to launch Pacemaker at startup. The use of Corosync means that all configured nodes in the cluster are constantly aware of each other s state and in the event of any service failure on one node, the other is informed. Pacemaker Pacemaker is described as a Cluster Resource Manager. Using Corosync as its cluster infrastructure, it monitors the cluster resources and nodes for any failures and will attempt to recover should a failure occur. Pacemaker itself consists of four internal components: CIB (Cluster Information Base); this maintains the current configuration and state of the cluster resources across the cluster. It uses XML and is constantly synched across all nodes. The PEngine component of Pacemaker uses the CIB to calculate the ideal state of the cluster and how this can be achieved. CRMd (Cluster Resource Management daemon); one node in the cluster will be designated as a DC (Designated Coordinator). The DC will act as a master node in the cluster responsible for overall management and decisions of the cluster itself. Should any failure on the DC occur, a new DC will be elected. The DC passes instructions handed to it by the PEngine to the other nodes in the cluster, who in turn report back with the results of those instructions. The PEngine will then take any action required based on the results returned. PEngine (Policy Engine); the Policy Engine acts as the brain of Pacemaker and controls how the cluster is to be managed, via the DC. All instructions and commands are processed and provided by the PEngine based on its internal calculations of the current cluster state. McAfee SIEM HA Receivers Product Guide 13

14 High Availability Components Pacemaker STONITHd; this is an acronym that stands for Shoot The Other Node In The Head. There may be a requirement to either reboot or shutdown a cluster node to prevent it interfering with the cluster as a whole due to a problem or failure. This task is handed off to STONITH to perform and is commonly referred to as Fencing. In our receiver HA configuration, the STONITH daemon uses IPMI to perform this task. We can check the current status and configuration of the CIB as follows; McAfee1-ERC-1250 ~ # crm status ============ Last updated: Mon Mar 30 07:32: Last change: Fri Mar 20 16:57: via crmd on McAfee2 Stack: openais Current DC: McAfee2 - partition with quorum Version: b d15cb702c9307df55512d323831a5e 2 Nodes configured, 2 expected votes 3 Resources configured. ============ Online: [ McAfee1 McAfee2 ] SharedIP (ocf::nitrosecurity:ipaddrmac): Started McAfee1 Clone Set: clonedipmi [FenceIPMI] Started: [ McAfee2 McAfee1 ] The above output reflects a cluster that is currently reporting as healthy. Any deviation from the above would indicate problems with the High Availability installation. The actual configuration can be viewed using the following command; McAfee1-ERC-1250 ~ # crm configure show node McAfee1 node McAfee2 primitive FenceIPMI stonith:external/nitrosecurity/ipmi \ params hostname="mcafee1 McAfee2" ipaddr=" " userid="root" passwd="mcafee123!" \ 14 McAfee SIEM HA Receivers Product Guide

15 High Availability Components IPMI op start interval="0" timeout="60s" primitive SharedIP ocf:nitrosecurity:ipaddrmac \ params ip=" " cidr_netmask="28" nic="eth1" mac="00:1e:67:8b:ea:a6" \ op start interval="0" timeout="60s" \ op stop interval="0" timeout="60s" clone clonedipmi FenceIPMI property $id="cib-bootstrap-options" \ expected-quorum-votes="2" \ no-quorum-policy="ignore" \ default-resource-stickiness="1000" \ stonith-timeout="2m" \ pe-error-series-max="100" \ pe-warn-series-max="100" \ pe-input-series-max="1000" \ stonith-enabled="true" \ dc-version="1.1.6-b d15cb702c9307df55512d323831a5e" \ cluster-infrastructure="openais" The above example illustrates the output expected if the cluster is healthy. Any deviation could suggest a problem with any of the two nodes or with the cluster itself. IPMI IPMI stands for Intelligent Platform Management Interface. It is an industry standard that allows for Out Of Band monitoring and administration of both software and hardware of an appliance. Usually implemented as part of the underlying BIOS and independent of the installed Operating System, it can be used for the remote management of the system as a whole to perform the following functions; Allow administration of a system that is powered down. Allow administration of a system independently of the installed OS (i.e. in the event of an OS failure). Remote management of the system hardware (i.e. temperature, component health, etc). Allow actions such as system power down or reboot to be performed remotely. When used as part of the Pacemaker clustering system, STONITH employs the reboot/shutdown facility of the IPMI interface to fence off or try to recover the node should it be required. We can check the IPMI configuration on the receivers using the ipmitool command, as follows: McAfee SIEM HA Receivers Product Guide 15

16 High Availability Components IPMI McAfee1-ERC-1250 ~ # ipmitool lan print 3 Set in Progress Auth Type Support Auth Type Enable : Set Complete : MD5 PASSWORD : Callback : MD5 PASSWORD : User : MD5 PASSWORD : Operator : MD5 PASSWORD : Admin : MD5 PASSWORD : OEM : IP Address Source : Static Address IP Address : Subnet Mask : MAC Address : 00:1e:67:8b:ea:ab SNMP Community String : public IP Header BMC ARP Control : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00 : ARP Responses Enabled, Gratuitous ARP Enabled Gratituous ARP Intrvl : 5.0 seconds Default Gateway IP : Default Gateway MAC : 00:00:00:00:00:00 Backup Gateway IP : Backup Gateway MAC : 00:00:00:00:00: q VLAN ID : Disabled 802.1q VLAN Priority : 0 RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12,0 Cipher Suite Priv Max : caaaxxaaaxxaaxx : X=Cipher Suite Unused : c=callback : u=user : o=operator : a=admin : O=OEM 16 McAfee SIEM HA Receivers Product Guide

17 Configuration Files High Availability Components Configuration Files Everything mentioned so far is configured and implemented using code and configuration developed specifically for use by our SIEM solution. There are a number of different configuration files that take part in the configuration and startup of the High Availability functionality; globals.conf ha.conf network.conf The globals.conf file holds the following HA specific entries; ha_enabled=yes ha_shared_mac=00:1e:67:8b:ea:a6 hb_network= other_iface="mgt; ; ; ;;off; ; " These are the settings as seen in the ESM web UI that relate to the HA configuration. The shared MAC entry is the one that will be utilized by the Data Source interface when it is enabled on the active receiver in the HA pair. The hb_network entry reflects one of the two networks available for selection in the UI for the Heartbeat interfaces. The other_iface entry holds the details of the Management Interface of the other receiver in the HA pair. The ha.conf is specific to the High Availability as its name suggests and should resemble the following: hi_bit=0 augmented=1 nodealias=rcvha1 o_nodealias=rcvha2 net= comm_addr= hwaddr=00:30:48:7b:30:8a hb_ipv6=fdb7:546d:06ee::2/64 o_hb_ipv4= ipmi=yes ipmi_ipv4= McAfee SIEM HA Receivers Product Guide 17

18 High Availability Components Configuration Files o_nodename=mcafee2 hb_ipv4= nodename=mcafee1 o_hb_ipv6=fdb7:546d:06ee::3/64 stonith_ipv4= ha_mode=primary Each of the settings above that are preceded with an o_ relate to the other receiver in the HA pair. In the above example the hi_bit setting is at 0, which means the ha.conf on the other receiver will have this setting configured at 1. The hwaddr setting in the ha.conf file is the MAC address that will be assigned to the eth1 interface used by the shared IP address when it is not currently active. The hb settings relate to the IP configuration for the ports used by the heartbeat while the ipmi_ipv4 setting relates to the interface that is directly attached to the IPMI interface on the other receiver. The stonith_ipv4 setting is the IP address that is configured on the IPMI lan channel. The ha_mode setting denotes the current status of the node. The network.conf file is used by all receivers whether configured as HA or standalone and is used to set the values used when configuring the network interfaces on the device. A minimal configuration for use in HA would be as follows: [global] accept_icmp_redirect=no generate_destination_unreachable=no [iface1.0] enabled=yes ipv4= ; ipv6= dhcp=no gateway= dns1= dns2= [iface0.0] enabled=yes ipv4= ; McAfee SIEM HA Receivers Product Guide

19 High Availability Components Configuration Files dhcp=no dns1= dns2= ipv6= gateway= [iface2.0] enabled=no ipv4= dhcp=no dns1= dns2= ipv6= gateway= [iface3.0] enabled=no dhcp=no dns1= dns2= ipv4= ipv6= gateway= [routes] McAfee SIEM HA Receivers Product Guide 19

20 High Availability Components Configuration Files In the above sample the interface listed as iface0.0 relates to the eth1 interface used by the shared IP address, while the iface1.0 relates to the management IP address. This same configuration would be seen on the other receiver in the pair but with a different IP address assigned to the management interface. As you can see in the above, the shared IP address is enabled and configured without any knowledge or thought as to whether this node is actually the active node in the pair and is also the same on the other node as well. This is by design and the assignment and activation of the shared IP is taken care of by other processes used in HA. However, you will also notice that there is no listing of the IP addresses used on the other interfaces required for HA to operate, i.e. The heartbeat and IPMI connections. This is because they are taken care of in the ha.conf file. It is possible to configure other available interfaces on the HA receivers as further management interfaces and in the sample above there are a further two available. 20 McAfee SIEM HA Receivers Product Guide

21 High Availability Setup Requirements 3 High Availability Setup The following chapter describes the steps required to configure two receivers in High Availability mode. Contents Requirements Cabling the Receivers Configuration Steps Requirements For High Availability to be configured there are a number of requirements that must be met: The two receivers must be cabled as described in the following section; Cabling the Receivers. The actual physical ports to use in relation to this will be dependent on the hardware models in use. Both HA receivers must be of the same model too. Three different IP addresses are required. o o o The shared IP address that will be used for the collection of data from the configured data sources. This refers to the IP address that is only ever present on the currently active receiver in the HA pair. The management IP address for the Primary receiver that will be used for communication with the Secondary Receiver, ESM and any configured ELM. It may also be used for direct SSH login to the receiver. This does not change between receivers. The management IP address for the Secondary receiver that will be used for communication with the Primary receiver, ESM and any configured ELM. It may also be used for direct SSH login to the receiver. This does not change between receivers. It is possible to use a receiver that is already installed to the ESM as the initial Primary receiver. This receiver may be utilized in a HA setup later with the introduction of a second receiver. It is also possible to use a new receiver with the express intention of immediate configuration for a HA setup. In either case, the Primary receiver must be installed to the ESM and the management interface initially configured with the selected Shared IP Address. During physical installation and initial setup, the secondary must be assigned the management IP address that has been selected for it and it must not be installed to the ESM at any point. NOTE: The Secondary receiver has ONLY the management IP address it is to be configured with as part of the HA configured during initial installation. DO NOT add the shared IP address as another interface IP address at any time. McAfee SIEM HA Receivers Product Guide 21

22 High Availability Setup Cabling the Receivers Cabling the Receivers The method by which the two receivers are cabled and the ports used will be dependent on the model in use and whether it is a 1U or 2U model. The following two diagrams illustrate how the two receivers must be cabled together. 1U Receivers cabling diagram 22 McAfee SIEM HA Receivers Product Guide

23 High Availability Setup Cabling the Receivers 2U Receivers cabling diagram McAfee SIEM HA Receivers Product Guide 23

24 High Availability Setup Configuration Steps Configuration Steps Configuration of the HA receivers via the ESM User Interface requires a number of steps to complete. The following assumes that the Primary receiver is already added to the ESM. 1 Configure the Primary receiver with the shared IP address and related network settings under the Network Interface Settings of the receiver already installed to the ESM. 24 McAfee SIEM HA Receivers Product Guide

25 High Availability Setup Configuration Steps 2 Navigate to the HA Receiver tab and enable the option Setup High Availability. You may also configure the Shared MAC Address value, although it is recommended to leave this as the default. The Heart Beat Network IP setting has two possible network values. o o These networks are used by the Heartbeat functionality, which is an entirely closed network due to the directly cabled connections and does not interact with the wider network in any way. However, two values are provided as a selection should one of the networks be utilized as part of the management or shared IP interfaces. McAfee SIEM HA Receivers Product Guide 25

26 High Availability Setup Configuration Steps 3 Configure the management IP address and related network settings for the Primary receiver. 26 McAfee SIEM HA Receivers Product Guide

27 High Availability Setup Configuration Steps 4 Configure the management IP address and related network settings for the Secondary receiver. McAfee SIEM HA Receivers Product Guide 27

28 High Availability Setup Configuration Steps 5 Having completed steps 1 4, if you return to the Network tab you will see that the initial configured management IP address field where the shared IP address was entered has now changed from being labelled as Management to Shared. Click the OK button to apply the new HA configuration. 28 McAfee SIEM HA Receivers Product Guide

29 High Availability Setup Configuration Steps 6 Once the OK button has been applied, a prompt appears requesting the password be entered to allow the Secondary receiver to be keyed. Enter the chosen password and click OK. The settings will then be applied and a number of notifications will be presented during the process in the pop-up dialogue box until it states the configuration has completed. McAfee SIEM HA Receivers Product Guide 29

30 High Availability Setup Configuration Steps 7 Once final confirmation that the settings have been applied return to the Receiver Information section to confirm that the HA is now operational and it should resemble the following. 30 McAfee SIEM HA Receivers Product Guide

31 High Availability Tools The User Interface 4 High Availability Tools A number of tools are provided that enable the administration of the High Availability on the receivers. Contents The User Interface The Command Line Interface The User Interface It is possible to administer the High Availability via the browser based User Interface to fulfill a number of tasks. The general status of the High Availability on both of the receivers can be found in the Receiver Information section of the Receiver Properties. This will provide a general overview of the status and report any service failures or issues with the High Availability and the receivers themselves. Administration of the High Availability via the User Interface can be found in two locations under the Receiver Properties. Network Interface Settings HA Receiver Reinitialize Secondary This can be used in the event that the Secondary receiver is replaced due to a failure. By configuring the new Secondary receiver with identical network related settings at installation this facility allows the new receiver to be introduced into the HA configuration. It will perform the HA configuration on the new receiver and apply the relevant settings related to both the HA and the configured data sources. High Availability Fail-Over This will perform a fail-over and swap the functionality of the receivers so that the currently active Primary attains Secondary status and vice-versa. This can be utilized on occasions where an upgrade is being performed or other maintenance is required on one of the receivers that might currently be acting in the Primary role. It allows minimal disruption to event collection by switching roles, thereby ensuring continuity of data and freeing the other sensor to be worked on without the risk of a loss of events. Standby By placing a receiver into Standby you are removing it as an active member of the HA. It is still a member of the High Availability but will not take an active part until returned to service. This is useful if it is under investigation for any issues that might arise. Return to Service As it suggests, this will reintroduce the receiver as an active member of the High Availability should it be reporting as offline or in standby. McAfee SIEM HA Receivers Product Guide 31

32 High Availability Tools The Command Line Interface The Command Line Interface There are two main commands that may be utilized at the CLI to check the status and administer the High Availability functionality on both receivers in the HA setup. ha_status This can be used to check the current status of the HA on a particular receiver. It is also vital for troubleshooting (which will be covered in a later chapter: Troubleshooting). An example of running ha_status on the Primary node, that returns a positive result from a healthy HA pair is as follows: McAfee1-ERC-1250 ~ # ha_status Ok hostname=mcafee1 mode=primary McAfee1=online McAfee2=online sharedip=mcafee1 stonith=none corosync=running hi_bit=no Reports the overall status Ok or NotOk The hostname of the receiver the command is run on The current mode of the receiver The status of receiver designated as McAfee1 The status of receiver designated as McAfee2 The receiver currently in possession of the shared IP This can be ignored. It plays no part in the status report The current status of corosync This receiver hi_bit assignment. Any deviation from the above would indicate a possible issue that requires investigation and remediation. The same command run on the Secondary node should return as follows: 32 McAfee SIEM HA Receivers Product Guide

33 High Availability Tools The Command Line Interface McAfee2-ERC-1250 ~ # ha_status Ok hostname=mcafee2 mode=secondary McAfee2=online McAfee1=online sharedip=stopped stonith=none corosync=running hi_bit=yes Reports the overall status Ok or NotOk The hostname of the receiver the command is run on The current mode of the receiver The status of receiver designated as McAfee2 The status of receiver designated as McAfee1 The receiver currently in possession of the shared IP This can be ignored. It plays no part in the status report The current status of corosync This receiver hi_bit assignment. ha_statuschange This can be used to control aspects of the receivers status as part of the HA pair. The command comes with a number of available switches that will control the receiver they are run on as follows: o o o o o ha_statuschange --standby This allows the receiver to be placed into a standby state. ha_statuschange --online If a receiver is reporting as offline or in standby this will bring it back online. ha_statuschange --noipmi This disables the IPMI functionality as used by STONITH. This is useful if you are investigating and troubleshooting the receiver and do not want STONITH to force a reboot as a result of that. ha_statuschange --ipmi This enables the IPMI functionality if it has been disabled. ha_statuschange --migrate If the receiver is currently reporting as Primary, this command will transfer the shared IP to the other receiver in the HA pair and place the receiver into Secondary mode. McAfee SIEM HA Receivers Product Guide 33

34 Troubleshooting Causes of Failover 5 Troubleshooting When troubleshooting High Availability receivers that are experiencing problems in relation to the HA functionality itself it is possible to use a number a number of approaches. Contents Causes of Failover Using the configuration files to locate the issue Scenarios that would require a manual return to an online status Causes of Failover Fail-overs have many, many causes. Some are caused by deliberate user action like during an upgrade. Others are caused by environmental issues like loss of power. Some are consequences from user actions like running NitroStop from the primary receiver's command line. The major cause of automatic failovers is the failure of heartbeats to arrive in the expected time frame. This can be caused by network congestion, bad wiring, bad NICs, bad switches, bad cables, or a receiver that is too busy. Determining the cause of the lost heartbeat can be a difficult endeavor. Other failure causes include a full hard disk and database that fails to run. It is impossible to cover every situation it is possible to apply a general method for finding a cause of a HA fail-over. The three most common scenarios of HA failure are: 1. User initiated fail-over 2. Loss of Power 3. Loss of communication The steps used to determine the cause of fail-over are: 1. Determine the time 2. Determine any immediately preceding actions and look for corresponding log entries 3. Look through all logs for the time frame immediately preceding the fail-over Log analysis is going to be the most important step at this stage. Unfortunately, logs can be difficult to read to determine a cause of HA failure if there are other issues that are filling the logs such as WMI failures. 34 McAfee SIEM HA Receivers Product Guide

35 Troubleshooting Causes of Failover User Initiated Fail-Over, Determine the Time The time of that fail-over can be determined from /var/log/messages. McAfee1-ERC-2250 ~ # grep restart /var/log/messages* /var/log/messages.1:feb 24 20:36:07 McAfee1 syslogd 1.5.0: restart. /var/log/messages.2:feb 21 18:56:35 McAfee1 syslogd 1.5.0: restart. User Initiated Fail-Over, Preceding Actions Here there are two restarts. One on Feb 21 and one on Feb 24. Looking in /var/log/messages.2 at the time just before the restart there is the following: Feb 21 18:54:45 McAfee1 hactl[6224]: caught SIGTERM(15) from '(unknown)' Feb 21 18:54:46 McAfee1 bootlog: Stopping Receiver High Availability...[ OK ] Feb 21 18:56:35 McAfee1 syslogd 1.5.0: restart. Looking back minutes before the restart there are no obvious entries to indicate what stopped hactl. Feb 21 18:44:13 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:45:18 McAfee1 last message repeated 2 times Feb 21 18:45:20 McAfee1 IPSDBServer[2648]: Resources: (7s, 6e, 122t, 148r, 2198f, 0q, 50p, [289 KB, 0 B, 3.18 MB], [32.5 MB, 21.4 MB, 9637]h, [1.64 MB, 488 KB, 39]bt, [44.0 KB, 40.4 KB, 5]bs, [864 KB, 79.2 KB, 451]tr, [3.50 MB, 2.75 MB, 15]sq, [2.41 MB, 771 KB, 8900]fp, [NitroInline: 1.76 MB, 1.49 MB, 1.52 MB, 30/872 KB, 598 KB, 629 KB, 30], [libelm: 84.9 MB, 84.6 MB, 84.6 MB, 23/84.9 MB, 84.6 MB, 84.6 MB, 23], [IPSDBServer: 584 KB, 287 KB, 309 KB, 18/520 KB, 287 KB, 308 KB, 17]) Feb 21 18:45:39 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:46:23 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:47:14 McAfee1 IPSDBServerctl[2622]: Info: -- Mark Feb 21 18:47:28 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:47:42 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:48:33 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:49:38 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:50:43 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:51:48 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:52:53 McAfee1 last message repeated 2 times McAfee SIEM HA Receivers Product Guide 35

36 Troubleshooting Causes of Failover Feb 21 18:53:58 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Feb 21 18:54:40 McAfee1 kernel: [ ] ADDRCONF(NETDEV_UP): eth1: link is not ready Feb 21 18:54:42 McAfee1 ntpd[2702]: Deleting interface #73 eth1, fe80::21e:67ff:fe51:97a3#123, interface stats: received=0, sent=0, dropped=0, active_time=2886 secs Feb 21 18:54:42 McAfee1 ntpd[2702]: Deleting interface #72 eth1, #123, interface stats: received=0, sent=0, dropped=0, active_time=2892 secs Feb 21 18:54:42 McAfee1 ntpd[2702]: peers refreshed Feb 21 18:54:42 McAfee1 bootlog: Stopping vaded...[ OK ] Feb 21 18:54:45 McAfee1 kernel: [ ] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Feb 21 18:54:45 McAfee1 kernel: [ ] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready Checking the other logs for items relating to hactl (do not grep /var/log/authfifo, grep will never return). grep hactl /var/log/[b-z]* grep 'Feb 21' /var/log/collectorsctl.log.06:feb 21 18:54:45 L_INFO caught SIGHUP(1) from 6224 'hactl' /var/log/collectorsctl.log.07:feb 21 19:02:46 L_INFO caught SIGHUP(1) from 2573 'hactl' /var/log/hasctl.log.06:feb 21 18:56:59 L_INFO (s) status-drop-path =/var/log/hactl.status /var/log/hasctl.log.06:feb 21 19:04:48 L_INFO (s) status-drop-path =/var/log/hactl.status /var/log/messages.2:feb 21 18:54:45 McAfee1 hactl[6224]: caught SIGTERM(15) from '(unknown)' /var/log/messages.2:feb 21 18:57:00 McAfee1 hactl[2573]: started /var/log/messages.2:feb 21 19:02:46 McAfee1 hactl[2573]: caught SIGTERM(15) from 5591 '(unknown)' /var/log/messages.2:feb 21 19:04:48 McAfee1 hactl[5838]: started /var/log/rcvha.log:fri Feb 21 18:56:59 UTC 2014 rcvha started hactl /var/log/rcvha.log:fri Feb 21 19:04:48 UTC 2014 rcvha started hactl grep hactl /var/log/[a-z]* grep 'Feb 21' /var/log/nitrovalidate.log.26:feb 21 16:27:29 L_INFO NOTE:(unknown) unknown file : /var/lock/hactl.lock /var/log/nitrovalidate.log.26:feb 21 18:58:07 L_INFO NOTE:(unknown) unknown file : /var/lock/hactl.lock 36 McAfee SIEM HA Receivers Product Guide

37 Troubleshooting Causes of Failover User Initiated Fail-Over, Immediate Time Frame Analysis can be performed for the 15 minutes before the restart. Note that when searching for days 1-9, they may appears as Feb 1, Feb 01, or Feb 1 (two spaces). egrep 'Feb 21 18:[45]' /var/log/[a-z]* less egrep 'Feb 21 18:[45]' /var/log/[b-z]* less produced such gems as (out of 2565 lines) /var/log/ipaddrmac.log:fri Feb 21 18:54:41 UTC 2014 IPaddrMAC wrote ha_mode=secondary to ha.conf /var/log/ipaddrmac.log:fri Feb 21 18:54:41 UTC 2014 IPaddrMAC sent SIGHUP to collectorsctl /var/log/collectorsctl.log.06:feb 21 18:54:45 L_WARN ha-mode: clean_standby /var/log/ha_corosyncmon.log:feb 21 18:54:45 INFO received TERM /var/log/ha_corosyncmon.log:feb 21 18:54:45 INFO attempting to stop corosync... /var/log/ha_corosyncmon.log:feb 21 18:54:45 INFO corosync stopped /var/log/ha_statuschange.log.3:feb 21 18:54:40 INFO ha_statuschange called by rcvha with option standby This validates the idea that the receiver went down peacefully. Grep does not search the corosync logs themselves, since they are compressed because of size. To search the corosync logs it is necessary to determine if any of them are applicable. Also note that the corosync logs from the DC are much busier than the corosync logs from the other receiver. ll /var/log/corosync.log* -rw-rw root root Mar 14 16:18 /var/log/corosync.log -rw-r root root Mar 14 04:02 /var/log/corosync.log.0.gz -rw-r root root Mar 13 04:02 /var/log/corosync.log.1.gz -rw-r root root Mar 12 04:02 /var/log/corosync.log.2.gz -rw-r root root Mar 11 04:02 /var/log/corosync.log.3.gz -rw-r root root Mar 10 04:02 /var/log/corosync.log.4.gz -rw-r root root Mar 9 04:02 /var/log/corosync.log.5.gz -rw-r root root Mar 8 04:02 /var/log/corosync.log.6.gz -rw-r root root Mar 7 04:02 /var/log/corosync.log.7.gz -rw-r root root Mar 6 04:02 /var/log/corosync.log.8.gz -rw-r root root Mar 5 04:02 /var/log/corosync.log.9.gz McAfee SIEM HA Receivers Product Guide 37

38 Troubleshooting Causes of Failover In this case, none are applicable since the earliest log is March 5. That would only have data going back to March 4. To search a compressed log you can use zcat. zcat /var/log/corosync.log.*.gz egrep 'Feb 21 18:[45]' less Loss of Power, Determine the Time Again determine the time of fail-over. McAfee1-ERC-2250 ~ # grep restart /var/log/messages Mar 14 16:37:58 McAfee1 syslogd 1.5.0: restart. Loss of Power, Preceding Actions Looking in /var/log/messages immediately before the restart it is evident that this receiver was running normally. Mar 14 16:30:00 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Mar 14 16:31:05 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Mar 14 16:32:10 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Mar 14 16:33:15 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Mar 14 16:34:20 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 Mar 14 16:37:58 McAfee1 syslogd 1.5.0: restart. As is often the case there is no preceding actions, everything was running the way it had been for days. Loss of Power, Immediate Time Frame Analysis of the logs to take place in the 15 minutes before the restart. There are 2912 lines in the logs for those minutes. egrep 'Mar 14 16:[23]' /var/log/[b-z]* less All the activities appear normal. /var/log/collectorsctl.log.07:mar 14 16:33:20 L_INFO caught SIGUSR1(10) from 2556 'healthmon.pl' 38 McAfee SIEM HA Receivers Product Guide

39 Troubleshooting Causes of Failover /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: do_pe_invoke_callback: Invoking the PE: query=2461, ref=pe_calc-dc , seq=179720, quorate=1 /var/log/corosync.log:mar 14 16:33:59 McAfee1 pengine: [2767]: notice: unpack_config: On loss of CCM Quorum: Ignore /var/log/corosync.log:mar 14 16:33:59 McAfee1 pengine: [2767]: notice: LogActions: Leave SharedIP (Started McAfee1) /var/log/corosync.log:mar 14 16:33:59 McAfee1 pengine: [2767]: notice: LogActions: Leave FenceIPMI:0 (Started McAfee1) /var/log/corosync.log:mar 14 16:33:59 McAfee1 pengine: [2767]: notice: LogActions: Leave FenceIPMI:1 (Started McAfee2) /var/log/corosync.log:mar 14 16:33:59 McAfee1 pengine: [2767]: notice: process_pe_message: Transition 1750: PEngine Input stored in: /var/lib/pengine/pe-input-45.bz2 /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=i_pe_success cause=c_ipc_message origin=handle_response ] /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: unpack_graph: Unpacked transition 1750: 0 actions in 0 synapses /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: do_te_invoke: Processing graph 1750 (ref=pe_calc-dc ) derived from /var/lib/pengine/pe-input-45.bz2 /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: run_graph: ==================================================== /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: notice: run_graph: Transition 1750 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input- 45.bz2): Complete /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: te_graph_trigger: Transition 1750 is now complete /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: notify_crmd: Transition 1750 status: done - <null> /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=i_te_success cause=c_fsa_internal origin=notify_crmd ] /var/log/corosync.log:mar 14 16:33:59 McAfee1 crmd: [2768]: info: do_state_transition: Starting PEngine Recheck Timer /var/log/filtersctl.log.07:mar 14 16:33:20 L_INFO caught SIGUSR1(10) from 2556 'healthmon.pl' /var/log/hasctl.log.07:mar 14 16:33:21 L_INFO caught SIGUSR1(10) from 2556 'healthmon.pl' /var/log/parsersctl.log.07:mar 14 16:33:21 L_INFO caught SIGUSR1(10) from 2556 'healthmon.pl' The above example shows entries from corosync.log. Any sudden failure in the receiver would resemble the same. It was running and then it wasn't running with nothing out of the ordinary in the logs. Sudden failures can include power failures, motherboard failures, catastrophic disk crashes, kernel panics and many more. McAfee SIEM HA Receivers Product Guide 39

40 Troubleshooting Causes of Failover To investigate possible hardware related failures the sel (System Event Log) on the ipmi chip can be used. Sometimes this may show up in /var/log/sel.log. The sel.log gets updated from the sel list on a normal shutdown. McAfee1-ERC-2250 ~ # ipmitool sel list 1 03/14/ :02:03 Event Logging Disabled #0x07 Log area reset/cleared Asserted 2 03/14/ :35:25 Power Unit #0x01 Failure detected Asserted 3 03/14/ :35:25 Power Unit #0x01 Power off/down Asserted 4 03/14/ :36:12 Power Unit #0x01 AC lost Asserted 5 03/14/ :36:12 Button #0x09 Power Button pressed Asserted 6 03/14/ :36:12 Processor #0x70 Presence detected Asserted 7 03/14/ :36:12 System Event #0x83 Timestamp Clock Sync Asserted 8 03/14/ :36:12 System Event #0x83 Timestamp Clock Sync Asserted 9 03/14/ :36:13 Power Unit #0x01 AC lost Deasserted a 03/14/ :36:14 Microcontroller/Coprocessor #0x16 Transition to Running b 03/14/ :36:14 Power Supply #0x66 Presence detected Asserted c 03/14/ :36:34 System Event #0x83 OEM System boot event Asserted Looking at the same time frame on the other receiver we see a transition in /var/log/corosync.log from: /var/log/corosync.log:mar 14 16:06:13 McAfee2 cib: [899]: info: cib_stats: Processed 19 operations (526.00us average, 0% utilization) in the last 10min /var/log/corosync.log:mar 14 16:16:13 McAfee2 cib: [899]: info: cib_stats: Processed 18 operations (0.00us average, 0% utilization) in the last 10min /var/log/corosync.log:mar 14 16:26:13 McAfee2 cib: [899]: info: cib_stats: Processed 18 operations (0.00us average, 0% utilization) in the last 10min To: /var/log/corosync.log:mar 14 16:36:10 corosync [pcmk ] info: pcmk_peer_update: memb: McAfee /var/log/corosync.log:mar 14 16:36:10 corosync [pcmk ] info: pcmk_peer_update: MEMB: McAfee /var/log/corosync.log:mar 14 16:36:10 McAfee2 crmd: [903]: notice: ais_dispatch_message: Membership : quorum lost 40 McAfee SIEM HA Receivers Product Guide

41 Troubleshooting Causes of Failover /var/log/corosync.log:mar 14 16:36:10 McAfee2 crmd: [903]: info: ais_status_callback: status: McAfee1 is now lost (was member) /var/log/corosync.log:mar 14 16:36:10 McAfee2 crmd: [903]: info: crm_update_peer: Node McAfee1: id= state=lost (new) addr=r(0) ip( ) r(1) ip( ) votes=1 born= seen= proc= /var/log/corosync.log:mar 14 16:36:10 McAfee2 cib: [899]: info: crm_update_peer: Node McAfee1: id= state=lost (new) addr=r(0) ip( ) r(1) ip( ) votes=1 born= seen= proc= /var/log/corosync.log:mar 14 16:36:10 McAfee2 crmd: [903]: WARN: check_dead_member: Our DC node (McAfee1) left the cluster /var/log/corosync.log:mar 14 16:36:10 McAfee2 crmd: [903]: info: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=i_election cause=c_fsa_internal origin=check_dead_member ] This transition took place because the receivers switched from non-dc to DC, which isn't a guarantee. Also, the corosync.log indicates McAfee1 is now lost, which indicates that some decisions on what to do with McAfee1 is forthcoming. /var/log/corosync.log:mar 14 16:36:10 McAfee2 stonith-ng: [898]: info: initiate_remote_stonith_op: Initiating remote operation reboot for McAfee1: e453d3a a8f034eb001 The above entry denotes that McAfee2 initiated a reboot on McAfee1. This is a stonith (Shoot The Other Node In The Head) and is used to put McAfee2 in a safe place to assume the SharedIP/SharedMAC. There will be a lot of log entries about deciding and implementing the stonith action. Also on the other receiver, /var/log/ipaddrmac.log will show the transition to primary. /var/log/ipaddrmac.log:fri Mar 14 16:36:13 UTC 2014 IPaddrMAC wrote ha_mode=primary to ha.conf Loss of Communication, Determine the Time grep restart /var/log/messages Mar 14 17:32:46 McAfee1 syslogd 1.5.0: restart. Loss of Communication, Preceding Actions The following shows problems with the eth4 interface: Mar 14 17:27:41 McAfee1 bootlog: nitrodbserver.init DB running. STARTED=1 STOPPING=0 McAfee SIEM HA Receivers Product Guide 41