Security Correlation Server Backup and Recovery Guide

Transcription

1 CorreLog Security Correlation Server Backup and Recovery Guide This guide provides information to assist administrators and operators with backing up the configuration and archive data of the CorreLog server, and restoring this data. The information presented here will be useful to administrators responsible for maintaining the CorreLog system, as well as backup operators, developers, and IT managers, with application in disaster recovery and other areas. CorrreLog Server provides various options to retain backup copies of files and configuration data in an online state. Additionally, CorreLog provides flexible elements that can be configured by a user to backup log data on the system to removable media, including the ability to execute periodic backup scripts, and log the success or failure of these operations for further analysis. These features are described here, to permit easy integration with existing backup software and enterprise operations. Backup and Recovery of Configuration Data CorreLog maintains its primary configuration data in the "CorreLog\config" directory of the system within a series of files that have a ".cnf" extension. The user can backup almost all of the configuration data of the system by simply backing up this single folder. In addition to the "config" directory above, several other locations and folders exist that contain configuration data requiring periodic backup. The following folders contain configuration data items that should be routinely backed up for data storage. Generally, these directories contain a small amount of data, and are easily backed up.

2 Note that files can be copied while CorreLog executes, and backup of these files does not require CorreLog to be stopped. CorreLog\actions Folder This folder contains the "Correlation > Actions" scripts of the system. This folder should be routinely backed up to capture any special files or modifications to existing scripts. The folder typically contains less than 100 Kbytes of data. CorreLog\c-alerts Folder This folder contains the "Alerts > Custom" scripts of the system. This folder should be routinely backed up to capture any special files or modifications to existing scripts. The folder typically contains less than 100 Kbytes of data. CorreLog\config Folder This folder (and all of its subfolders) contains the main configuration data of the system, including templates, installation defaults, and checkpoints. This folder contains virtually all the configuration data of the system (with a few notable exceptions, described here). The folder typically contains less than 1 Mbytes of data. CorreLog\dash Folder This folder (and all of its subfolders) contains the configuration items associated with the dashboard facility, which works with the data in the "CorreLog\config" folder listed above. This folder typically contains less than 100 Kbytes of data. CorreLog\graph Folder This folder contains graph configuration files for use with the CorreLog graph viewer. These files are normally edited via the Graph Screen. The folder typically contains less than 100 Kbytes of data. CorreLog\t-actions Folder This folder contains the "Correlation > Ticket Actions" scripts of the system. This folder should be routinely backed up to capture any special files or modifications to existing scripts. The folder typically contains less than 100 Kbytes of data. CorreLog\tickets Folder This folder contains the ticket data of the system scripts. This folder should be routinely backed up to capture ticket information on the system. The folder typically contains less than 10 Mbytes of data. Note that CorreLog does not need to be stopped in order to backup the above files. However, to restore any file within these directories, CorreLog should be first stopped, and then restarted. (This is because the above files are read by CorreLog processes on service startup, and then periodically overwritten with new data.) Backup and Recovery Guide, Page 2

3 Special Configuration Data Recovery Functions CorreLog provides two other functions to allow rapid restoration of configuration data (as an alternative to restoring a backup of data, as explained above.) These two functions allow the user to quickly restore a particular configuration file, useful if a configuration change needs to be eliminated or reverted to the previous day. These features are explained below in more detail. "Yesterday" Template File. As a standard function, CorreLog will automatically store configuration data items associated with the Correlation Engine each night at midnight. This data is stored in a special "Template" file called "Yesterday", which can be loaded by the operator via the "Correlation > Config > Templates" facility. This allows the operator to quickly revert the "Threads", "Alerts", "Macros", "Address Groups" and other items to their previous version, which existed at midnight. Checkpoint Files. CorreLog creates a checkpoint of all configuration data and status data each night, and stores this data in the "CorreLog\config\$chkpt" directory of the system. Each night, at midnight, all configuration files are copied to this directory, and appended the day of the week (where 0=Sunday and 6=Saturday.) To restore these files, the user must (1) locate the particular configuration file in the CorreLog\config\$chkpt" directory; (2) stop the CorreLog Server; (3) copy the file to the correct location in the "config" directory and; (4) restart the CorreLog server. Backup of Log Data And Archives The vast majority of CorreLog operational data resides in the "log" and "archive" folders of the system. This data can constitute an enormous amount of data (potentially Terabytes of log data) so the backup of this data is an important consideration. CorreLog provides several techniques to accomplish this, described here. Direct Backup Of Log Data The user can perform an incremental backup of the "archive" and "logs" directory to tape or other media, using a variety of commercial software. In this case, an important consideration is to realize that only newly logs should be backed up, to conserve storage. (Otherwise, a full backup may exceed the capacity of most backup systems.) Forwarding Syslog Messages to Other Receiver The administrator can forward log files to a second copy of the CorreLog server that exists strictly for this purpose. In this case, the user configures the master CorreLog server to forward all log messages using the "Messages > Config > Backup and Recovery Guide, Page 3

4 Forwarding" screen of the system. The backup copy of CorreLog can also serve as a "hot swappable" version of the system. Schedule Copy or FTP Push of Archives The administrator can create a simple script that pushes the archive data to another server (possibly a UNIX server, or other windows server) at nightly intervals. This script can be executed via the CorreLog "System > Scheduler" function to perform a daily backup, weekly backup, or monthly backup. Restoration of Log Data Restoration of log data (from an archive file, or system backup) is not particular difficult, but it may be required to re-index this data and regenerate catalogs of this data to fully restore the entire configuration of the CorreLog Server. Restoration of Log Data The user can restore log data to the system from an archive file by simply unzipping the archive into the CorreLog "logs" directory. (The location of the logs data directory is configured on the "Messages > Config > Parms" screen.) The Administrator can execute the "RestoreArc.exe" program (documented below) to simplify this operation. Restoration of Catalog Data Restoration of log data does not necessarily rebuild all the catalogs with this information. To restore the catalog data, the user may have to re-index and regenerate all the catalogs using the "RestoreArc.exe -index" option or the "CGenx.exe" program (in the "cli-bin" directory.) These commands are documented in more detail within the next several sections. Note that, when restoring an archive file, the operator can first search for the appropriate archive via the "Reports > Query" function, which allows the user to search compressed archive files on the system. Once the archive file is located, the operator simply executes "RestoreArc.exe" to place the contents of the compressed file into the "logs" directory. Restoring Archive Data With "RestoreArc.exe" The easiest way to restore a single archive file for a specific date is to use the "RestoreArc.exe" program, which is a standard utility found in the "system" folder of the CorreLog Server. This utility unzips the archive, then re-indexes the keyword list, and optionally re-indexes all the various other index files, used by the "Devices", "Users", "Facilities", "Severities", and "Threads" screens. Backup and Recovery Guide, Page 4

5 To restore a log file from an archive file, the user creates an administrative command prompt, and issues the following command in the "CorreLog\system" directory. (The date below of is substituted for the date of the archive file.) RestoreArc.exe overwrite The above command unzips the archive file, and places it in the correct location in the "logs" directory. The command also re-indexes the keyword list. This enables the " Search", " Query", and the " Graph" screens to access the data. HOWEVER, this does not update any of the catalog data (i.e. the "Devices", "Users", "Facilities", "Severities", or "Threads" screen.) To make data accessible to these screens, also execute the following command. RestoreArc.exe index The above command can take some time to execute; hence it must be explicitly executed. (Depending on the size of the log file, the above command can take several minutes or even several hours to complete) Executing the "RestoreArc.exe" command with no arguments displays brief but complete help on the command usage and argument syntax. Repairing And Re-indexing Data With "CGenx.exe" As an alternative to executing the "RestoreArc.exe" program with the "-index" option, the administrator can also execute the "CGenx.exe" command line utility to rebuild index files. This command is documented in the "Command Line Interface (CLI) User Manual", which is available in the "s-doc" directory of each CorreLog installation. (The "RestoreArc.exe (date) index" command is an alias for the "CGenx.exe all (date)" command. The "CGenx.exe" program permits the operator to individually re-index any catalog on the system, and provides an equivalent (but slightly more flexible) interface than the "RestoreArc.exe" program. The "CGenx.exe" program is also useful for fixing corrupt index files that may cause "Unformatted Text" messages to appear on certain message list screens. Backup of Agent Configurations As a special note, the operator can backup the configuration files of agents using the "rsmconf.exe download" command line utility. This script downloads a configuration file from a specified agent that can later be uploaded to the same agent or some other agent to affect recovery functions. Backup and Recovery Guide, Page 5

6 Detailed notes on the "rsmconf.exe" utility are included in the "Windows Tool Set User Manual", Section 5. (This manual is available from the "Home" screen of the CorreLog Server, after user login.) The agent configuration files reside at the physical location of the managed platforms, and the data is not immediately available for backup at the CorreLog Server. If the CorreLog site has many strategic agent configurations, it may be useful to create a small batch file to backup these agents. For example, the user can create a simple batch file as follows that will copy the agent configurations to an Agent-Backup folder. This script could be launched via the CorreLog "Scheduler" routine at periodic intervals (such as each day.) REM: # BACKUP_AGENTS.bat REM: # Backup strategic CorreLog Agent configurations. if not exist..\agent-backup mkdir..\agent-backup rsmconf.exe download >..\Agent-Backup\1_1_1_1.bak rsmconf.exe download >..\Agent-Backup\1_1_1_2.bak rsmconf.exe download >..\Agent-Backup\1_1_1_3.bak rsmconf.exe download >..\Agent-Backup\1_1_1_4.bak echo Agent Backup completed. REM # Agent config backup complete. No further action is required. REM # Finished The above script can be edited to supply the list of strategic agents, and placed in the "system" directory of the CorreLog Server. Then, the script can be periodically executed (such as with the "Scheduler" function) to copy the agent configurations from remote agents to the local machine. The "Agent-Backup" folder will then retain the latest copies of the agent programs. Agent configurations can then be restored (if needed) by uploading the values back to the agents using the "rsmconf.exe upload" option (or using the "Directly Edit Remote Agent Config" link on the CorreLog Server platform.) The above script can be improved to first download the agent configuration to a temporary file, then copy the files into the "Agent-Backup" folder once all downloads are successful. (This provides a hedge against overwriting a backup with an empty file when the agent is not accessible for some reason.) Executing the "rsmconf.exe" program with no arguments provides brief but complete help on the usage of this program, which can also be used to check configuration data. Backup and Recovery Guide, Page 6

7 Example Configuration Data Backup Script An example configuration data backup script is provided below. The information below can be (1) cut and paste to a batch file; (2) modified by the administrator to adjust parameters; (3) placed in the "CorreLog\System" directory, and then (4) launched by the "System > Schedule" function of CorreLog. As written below, the configuration data backup script places the CorreLog system configuration data onto the "D\:" drive, in a directory with the same name as today's date. The script below can be modified by an administrator to change its operating characteristics. REM # BACKUP_CONFIG.bat REM # This script creates a folder on the "D:" drive with today's REM # date, and copies all config data of the system into that location REM # This is the location where files are stored: set BACKUP_DIRECTORY=D:\CorreLog_Config REM # Get today's date in YYYY-MM-DD format: FOR /F "tokens=1-4 delims=/ " %%I in ('date /t') do ( set FILEDATE=%%L-%%J-%%K ) set MAIN_DIRECTORY=%BACKUP_DIRECTORY%\%FILEDATE% REM # Make subdirectories: mkdir %MAIN_DIRECTORY%\actions mkdir %MAIN_DIRECTORY%\c-alerts mkdir %MAIN_DIRECTORY%\config mkdir %MAIN_DIRECTORY%\dash\config mkdir %MAIN_DIRECTORY%\dash\layouts mkdir %MAIN_DIRECTORY%\graph mkdir %MAIN_DIRECTORY%\t-action mkdir %MAIN_DIRECTORY%\tickets REM # Copy files here. copy..\actions\*.* %MAIN_DIRECTORY%\actions copy..\c-alerts\*.* %MAIN_DIRECTORY%\c-alerts copy..\config\*.* %MAIN_DIRECTORY%\config copy..\dash\config\*.* %MAIN_DIRECTORY%\dash\config copy..\dash\layouts\*.* %MAIN_DIRECTORY%\dash\layouts copy..\graph\*.* %MAIN_DIRECTORY%\graphs copy..\t-actions\*.* %MAIN_DIRECTORY%\t-actions copy..\tickets\*.* %MAIN_DIRECTORY%\tickets REM # Config backup complete. No further action is required. REM # Finished Backup and Recovery Guide, Page 7

8 Example Archive Data Backup Script An example archive data backup script is provided below. The information below can be (1) cut and paste to a batch file; (2) modified by the administrator to adjust parameters; (3) placed in the "CorreLog\System" directory, and then (4) launched by the "System > Schedule" function of CorreLog. As written below, the log data backup script copies CorreLog system archive data onto the "D\:" drive, in a directory with the same directory structure as the CorreLog archive folder. Only archives that have not been previously backed up are copied, to conserve disk space and CPU. The script below can be modified by an administrator to change its operating characteristics, such as to additionally backup message digest files, or the raw log file data of the system. REM # BACKUP_ARC.bat REM # This script backs up all the archive files to a location on REM # the D:\ drive configured by the operator below. The script REM # can be launched by the CorreLog "System > Schedule" function REM # daily basis to copy ONLY new archive files. REM # This is the location where files are stored: set ADIR=D:\CorreLog_Archive REM # Get today's year value: FOR /F "tokens=4 delims=/ " %%I in ('date /t') do set YEAR=%%I REM # Make the backup directory structure. mkdir %ADIR% mkdir %ADIR%\archive\ mkdir %ADIR%\archive\%YEAR% REM # Copy the archive files, only if they have not been previously REM # copied by this script. FOR %%I in (..\archive\%year%\*.gz ) do ( ) if NOT EXIST %ADIR%\archive\%%I COPY %%I %ADIR%\archive\%%I REM # Archive backup complete. No further action is required. REM # Finished. Backup and Recovery Guide, Page 8

9 Remote Configuration Backup via HTTP and ZIP It is fairly easy to configure the HTTP server and ZIP program to backup configuration data, initiated by a remote HTTP request. This technique may be useful for situations such as failover, disaster recovery, or simple remote storage of backups. The script below can be placed in the "x-cg\r-backup.bat" folder, and then an HTTP request (such as via the wget.exe utility) can generate the backup and then download the backup ZIP file. This requires permissions to be modified for the "x-cgi" folder via a modification to the "htaccess.txt" file in that directory. REM: # R-BACKUP.bat REM: # Backup a files to the BACKUP_FILE_NAME zip file REM: # This script can be launched via an HTTP request by REM: # Storing this file in the "CorreLog\x-cgi" folder, REM: # and modifying the "htaccess.txt" file in that folder. set BACKUP_FILE_NAME=..\s-html\BACKUP.zip REM: These are the folders to backup. ( echo../advisory echo../audit echo../c-alerts echo../config echo../dash echo../etext echo../excel echo../graph echo../net-snmp echo../net-user echo../query echo../s-html echo../t-actions echo../temp echo../u-agent )> ziplist.txt REM: # Zip the files here. del /f %BACKUP_FILE_NAME% type ziplist.txt..\system\zip.exe -r -@ %BACKUP_FILE_NAME% del /f ziplist.txt REM: # Finished. The BACKUP_FILE_NAME has been created. Note that the "x-cgi" CGI folder normally requires authentication to execute any program therein. The "htaccess.txt" file in that folder should be modified to permit the R- BACKUP.bat file above to be launched from a specific IP address or machine. The resulting BACKUP.zip file can then be downloaded as a separate operation. Backup and Recovery Guide, Page 9

10 Creating a Custom Configuration Package Using the techniques discussed in this manual, an operator can easily create a custom configuration package that can assist with deployment of the software at multiple sites. For example, if the CorreLog Server is to be installed and configured at may different locations, and a special target configuration exists, the operator can create a zip file (or other archive file) containing the desired configuration data of the system, and then unzip these files into the CorreLog Server root directory to effect an immediate configuration of the system. The configuration data of a site resides in specific folders, similar to the data previously identified, as follows: CorreLog\config Folder This folder (and all of its subfolders) contains the main configuration data of the system, including templates, installation defaults, and checkpoints. The user should include this directory in any custom package. CorreLog\actions Folder This folder contains the "Correlation > Actions" scripts of the system. The user may include these files in a custom package if there are one or more special "bat" programs referenced by the "Correlation Actions" screen. CorreLog\c-alerts Folder This folder contains the "Alerts > Custom" scripts of the system. The user may include these files in a custom configuration package if there are one or more special "bat" programs referenced by the "Custom Alerts" screen. CorreLog\dash Folder This folder (and all of its subfolders) contains the configuration items associated with the dashboard facility, which works with the data in the "CorreLog\config" folder listed above. The user may include these files in a custom package if there are any special dashboard elements. CorreLog\graph Folder This folder contains graph configuration files for use with the CorreLog graph viewer. The user may include these files in a custom package if there are any special graphs. CorreLog\t-actions Folder This folder contains the "Correlation > Ticket Actions" scripts of the system. The user may include these files in a custom package if there are one or more special "bat" files referenced by the "Ticket Actions" screen. Backup and Recovery Guide, Page 10

11 To create a custom package, the operator executes the following procedure. 1. The operator identifies a "Master" CorreLog site, and configures the CorreLog Server as desired, including dashboards, threads, alerts, and other necessary configuration data. 2. The operator creates a Zip file (or other archive file) containing the various directories and files identified at the start of this section. The resulting archive will be a relatively small package, typically less than 100 Kbytes in size, which is easily redistributed to other sites. 3. At a new site, the operator installs the COTS CorreLog Server software using the standard installation procedure documented in the "CorreLog Installation and Quick Start" manual. 4. After installation is complete, the operator stops the CorreLog Server Framework Service, extracts files from the package prepared in step (2) above, and then restarts the CorreLog Server. The above steps are sufficient to configure the CorreLog Server dashboards, threads, alerts, custom actions, and other items represented in the master site, with no other action necessary on the part of the end-user of the site. Note that variations to the above procedure are possible, such as including additional screens and documentation, or dynamic creation of threads and alerts based upon external data, and installation of plug-in components that may be necessary or desireable. Backup and Recovery Guide, Page 11

12 Professional Services CorreLog, Inc. provides professional services to assist in all phases of deployment activities, and assists with all deployment and operational issues through its support division. Prior to deploying CorreLog Server (whether a small single-tier deployment or a large multi-tier deployment) licensed users are encouraged to contact CorreLog for a detailed discussion of options and capabilities. Further information on installation is available from the "Home" page of each CorreLog installation, and is contained in the comprehensive documentation accompanying the product. Refer to the "Installation and Quick Start Guide" for additional information For Additional Help Detailed specifications regarding the CorreLog Server, add-on components, and resources are available from our corporate website. Test software may be downloaded for immediate evaluation. Additionally, CorreLog is pleased to support proof-ofconcepts, and provide technology proposals and demonstrations on request. CorreLog, Inc., a privately held corporation, has produced software and framework components used successfully by hundreds of government and private operations worldwide. We deliver security information and event management (SIEM) software, combined with deep correlation functions, and advanced security solutions. CorreLog markets its solutions directly and through partners. We are committed to advancing and redefining the state-of-art of system management, using open and standards-based protocols and methods. Visit our website today for more information. CorreLog, Inc. mailto:support@correlog.com Backup and Recovery Guide, Page 12