Users manual. GOL installation. Version 1.0

Transcription

1 Users manual GOL installation Version

2 Content 1 GOL and its components Description of golloader agents Description of goldb Description of GOL-GUI Requirements for GOL installation GOL installation sequence Preparing for GOL installation Installation of goldb, GOL-GUI and Linux golloader goldb installation and initial configuration GOL-GUI installation Linux golloader installation and initial configuration Linux golloader configuration file Installation and initial configuration of individual golloader on other Linux hosts Additional configurations for CentOS operating system Configuration of SNMP trap GOL configuration after installation Detection of goldb IP address Getting started with GOL Finishing the work with GOL Linux golloader configuration Initialization of new golloader agent Configuration of log data sources Installation and configuration of Windows golloader agent (WinGol) Preparing for WinGol installation Installation of WinGol WinGol configuration in GOL WinGol and Linux golloader uninstallation Preparing GOL prior WinGol uninstallation Linux golloader and GOL uninstallation Restore WinGol installation goldb management environment Access to the goldb user's environment Overview of the goldb server resources Database management Stop and activation of individual databases Deleting documents from database Commands that can be executed in database goldb users and passwords Clusterpark Ltd. 2. / 52

3 9.7 Database license and its extension Troubleshooting GOL GOL system folders Linux golloader agent is seen as inactive in GOL-GUI network-chart GOL-GUI stops after login GOL-GUI is not running List with available log types in golloader configuration window is empty Clusterpark Ltd. 3. / 52

4 List of pictures Picture 1. GOL access form Picture 2. Window with GOL configuration elements Picture 3. Example with new GOL golloader agents Picture 4. Window for configuration of golloader main settings Picture 5. Window for list of log data sources Picture 6. Log data source configuration window Picture 7. List of configured log data sources Picture 8. List with configured hosts Picture 9. WinGol installation starting window Picture 10. Window for input of host name and description Picture 11. Window for input of goldb connection settings Picture 12. Window for input of WinGol installation folder Picture 13. Dialog for approval of WinGol installation folder Picture 14. Window for input of folder where log data will be collected Picture 15. Window with overview of installation folders Picture 16. Installation status window Picture 17. Windows with list of newly installed loaders Picture 18. Window for configuration of WinGol main settings Picture 19. List of automatically recognized MS Windows events Picture 20. Log data transmission configuration window Picture 21. List with enabled for transmission data sources Picture 22. List with configured hosts Picture 23. List of devices registered in GOL Picture 24. Example of list with disconnected device Picture 25. Dialog window for approval of device removing Picture 26. List of hosts without deleted device Picture 27. List with deleted entry Picture 28. WinGol uninstallation dialog Picture 29. Window with message about successful WinGol uninstallation Picture 30. List with removed hosts Picture 31. Host configuration window Picture 32. Host connection configuration window Picture 33. Drop down list with available loaders Picture 34. goldb terminal user access form Picture 35. Overview of goldb server resources Picture 36. Database information window Picture 37. Example with changed values in goldb Picture 38. List with goldb users Picture 39. Window for change of goldb user password Clusterpark Ltd. 4. / 52

5 List of tables Table 1. Description goldb database instances Table 2. Description of golloader installation settings Table 3. Full description of golloader configuration settings Table 4. Description of golloader installation settings Table 5. Description of log data source settings Table 6. Examples for Linux log data sources Table 7. Examples of log file names as regular expressions Table 8. Description of input parameters for host name Table 9. Description of goldb connection settings Table 10. Description of loader configuration settings Table 11. Description of log data transmission settings Table 12. Commands available for goldb management Clusterpark Ltd. 5. / 52

6 1 GOL and its components GOL is Clusterpark developed powerful log data consolidation solution with open, XML standard, document oriented data base that provide very fast data analysis. With GOL can be detected the security breaches, software related and technical problems in IT infrastructure (computers, servers, network devices etc.). GOL can be used as complementary solution to other analysis tools for deeper understanding of processes in different systems for investigation and quick decisions for preventive activities. GOL has following components: golloader: agent for log entry collection from files and sending them to database (goldb) for consolidation from local network or within WAN. GOL installation has two agents: one for collection of logs from Linux based systems and another one for log data collection from devices with Windows operating systems; goldb: consolidated log entry database, that has been built on Clusterpoint XML document oriented NoSQL database engine; GOL-GUI user interface: GOL web oriented user interface for access of consolidated log data, search of information in them, data representation in graphical form and event monitoring in real-time with dashboards and alerts for crucial events; API: for integration of data from goldb with other analytics systems or creation of statistics reports with simple XML requests from applications like MS Excel. More information about API and how to use it can be found at Clusterpoint Wiki page. 1.1 Description of golloader agents golloader agents for Linux operating systems and WinGol for MS Windows OS are installed on devices (servers or computers) where the log data must be collected for transmission to goldb. Agents transmit log data to goldb within time intervals that can be configured centrally from GOL-GUI. They provide collection of actual and historical (rotated) log data. Besides the transmitting of log data agents contact the goldb at regular time intervals for receiving of configuration changes. 1.2 Description of goldb goldb provides indexing and consolidation of log data received from agents in a single database. Thanks to technologies built in the Clusterpark database engine goldb provides fast information search by using single words, phrases or complex queries with Boolean expressions. goldb can be clustered and mirrored thus building systems for ensuring additional data security and increased productivity. More about Clusterpark DB engine and its capabilities can be found at GOL has several database instances whose role is described in Table 1. Table 1. Description goldb database instances. Name of Description of database database goldb All log entries received from golloader agents are stored there. golcfg Contains information about all sorts of GOL configuration parameters. golloader settings are stored there as well. golstat There are stored the settings necessary for operation of dashboards and alerts. goldb has its own web oriented management system that allows configuration of clusters and monitoring of resources used. Clusterpark Ltd. 6. / 52

7 1.3 Description of GOL-GUI GOL-GUI provides access to information collected in goldb and functionality for its analysis: Time graphs for highly interactive, visual evaluation of log entry amounts within selected time periods for finding of periods with increased or decreased amounts of log entries that can indicate the potentially anomalous activity; Network graphs for visualization of all log sources, their cross-links and statuses; Broad possibilities for Google-like information search by using simple words, phrases, Boolean expressions and structured queries; Dashboards for real-time IT infrastructure and software monitoring with time graphs and selective search queries; Real-time alerts about crucial events by using selective search queries and set of rules for their identification; Preservation of search queries for later reuse of them; Centralized configuration and management of golloader agents; Configuration of GOL general settings. GOL-GUI is created by using HTML5 and it can be used with smart phones and tablets. Clusterpark Ltd. 7. / 52

8 2 Requirements for GOL installation GOL should be installed on Linux operating systems. 32 and 64 bit Linux OS is supported. Currently GOL could be installed on - Debian, Fedora, CentOS and Suse. GOL can be installed on Windows or Mac OS too. In this case the virtualization environment (like Oracle Virtual Box) with, supported by GOL, Linux OS installed must be used. MS Windows event log file agent can be installed on MS Windows 2000, Vista, XP, 7, 8 workstation and MS Windows Server operating systems. Hardware requirements for using GOL largely depend on amount of log files that should be consolidated and analyzed. Minimum requirement is to have at least 2 CPU cores, 4Gb RAM and 10Gb free disk space. goldb management system can be used for monitoring of resources and planning of them, taking into account the time how long the data should be stored. GOL trial can be used for evaluation of resources and functionality before purchase of licensed solution. Clusterpark Ltd. 8. / 52

9 3 GOL installation sequence GOL is prepared and available as Linux installation package. Package contains all components for GOL as described in chapter 1 GOL and its components excluding WinGol agent for Windows OS that could be installed separately. Recommended GOL installation sequence is following: Prepare for installation and download GOL installation package. Described in chapter 3.1 Preparing for GOL installation; Install goldb, GOL-GUI and Linux golloader. Described in chapter 3.2 Installation of goldb, GOL-GUI and Linux golloader; o perform additional configuration of the CentOS operating system if necessary. Described in chapter 3.4 Additional configurations for CentOS operating system; Configure SNMP trap. Described in chapter 3.5 Configuration of SNMP trap; Perform configuration of GOL settings. Described in chapter 4 GOL configuration after installation; Connect to GOL on goldb server installed golloader. Described in chapter 5 Linux golloader configuration. Step can be omitted if golloader on goldb server was not installed; Optional installation of other golloader Linux agents. Described in chapter 3.3 Installation and initial configuration of individual golloader on other Linux hosts; Optional installation of WinGol MS Windows agents. Described in chapter 6 Installation and configuration of Windows golloader agent (WinGol). 3.1 Preparing for GOL installation Before GOL installation the hardware or virtual machine with supported Linux OS must be prepared. During installation the SMTP server settings will be asked to enter (chapter goldb installation and initial configuration). Therefore it is recommended before installation to ascertain the SMTP server address. That can be done in several ways. For example: In case that on user computer the Outlook or Thunderbird is installed, than note sending settings; Ask for SMTP address the system administrator or internet service provider; It is highly recommended to assign static IP address to host where GOL installation is planned. In order to download GOL package in Linux command line can be entered command wget or curl and link to appropriate Linux package: Debian, Ubuntu, Suse: Use command wget; Clusterpark Ltd. 9. / 52

10 CentOs, RedHat, Fedora: Use command curl; 3.2 Installation of goldb, GOL-GUI and Linux golloader goldb installation and initial configuration After downloading of installation package go to the folder where package is stored. For installation of package use command sh or./: To continue goldb installation, on question install gol-gui package? enter YES, but for cancellation of installation process enter NO. After an authorization to begin the installation host will be checked for necessary components. All components necessary for GOL are included in installation package and if something will be found missing on host, it will be installed. To start component checking any key could be pressed. After checking of necessary components a list of goldb software setup and installation size will be displayed to user. Clusterpark Ltd. 10. / 52

11 To continue installation type y and press Enter ; To cancel goldb installation type n and press Enter. After goldb installation initial configuration must be done. In requested fields enter information as needed: Please enter system address GOL system address; Please enter SMTP server address SMTP server address; Please enter GOL administrator s address GOL administrator address; goldb installation and configuration is finished and in next step installation of GOL-GUI is offered GOL-GUI installation After checking of necessary components a list of GOL-GUI software setup and installation size will be displayed to user. To continue installation type y and press Enter ; To cancel GOL-GUI installation type n and press Enter. Clusterpark Ltd. 11. / 52

12 After installation of GOL-GUI additional configuration is not necessary and in next step installation of golloader is offered Linux golloader installation and initial configuration golloader agent can be installed on any Linux computer or server. It is recommended to install loader on goldb server as well for monitoring its Linux environment. To continue installation of loader on goldb server type Yes and press Enter ; If golloader installation on GolDB server is not necessary, type No and press Enter. If golloader installation on goldb server is confirmed, than after checking of necessary components, a list of golloader software setup and installation size will be displayed to user. To continue installation type y and press Enter ; To cancel golloader installation type n and press Enter. After installation of loader its initial configuration is needed. It is required to use already installed golloader parameters or configure them. It is recommended to use existing GOL-GUI settings by typing Yes and pressing Enter. Clusterpark Ltd. 12. / 52

13 Important settings installation will offer by default. Description of settings is available in Table 2. Table 2. Description of golloader installation settings. Settings field Clusterpoint Server Host? Clusterpoint Server Port? Loader name? () Loader description? () Input description By default (localhost). Input must be approved by pressing Enter. Specifies the network port through which golloader will send data. By default the port (5550) will be used. Input must be approved by pressing Enter. Must be specified the name of server or computer (host) on which the loader will be installed. This name will be used to identify host in goldb and GOL-GUI. Input must be confirmed by pressing Enter. In this field description of host can be written. This field is not mandatory. Input must be confirmed by pressing Enter. Clusterpark Ltd. 13. / 52

14 3.2.4 Linux golloader configuration file. If goldb server IP address has been changed or during golloader installation wrong settings had been entered, they can be changed directly in golloader configuration file. It is recommended to do changes in settings in following steps: 1) Login to Linux with root user rights by using commands su l or sudo s : 2) Stop the golloader service with command: /etc/init.d/gol-load stop : 3) Open configuration file in Linux text editor (vi, nano, gedit u.c.). Configuration file is located in folder: /usr/local/gol-loader/conf.ini: 4) Change the setting that was entered wrongly. Description of settings is available in Table 3. Table 3. Full description of golloader configuration settings. Settings field Settings description Automatically created golloader identification in goldb. Should Loader_id = not be changed. Name of host on which golloader was installed. This name is used for loader identification in goldb and GOL-GUI. Can be changed if name = necessary. description = Additional description of host. Can be changed if necessary. Specifies the number of records loader sends to goldb each maxrows = session. By default, it is 200 records. Clusterpark Ltd. 14. / 52

15 sleep = address = protocol = port = username = password = path = type = Logs transmission time interval. The default is 20 sec. goldb server IP address. A network protocol used for connection to goldb. By default, the - tcp. Specifies the network port through which golloader send data. Username for connection to goldb. Password for connection to goldb. The log data source configurations file name and location. Type of operating system linux or Windows. 5) Start golloader service by entering command: /etc/init.d/gol-load start Warning! If golloader already was configured in GOL-GUI before changes in configuration file, than as soon as golloader connects to the goldb, the values in fields name, discription, sleep and maxrows will be change according the configuration already saved in database. There fore changes in these fields directly in configuration file are not necessary and they can be changed in GOL-GUI Configuration section Hosts. Clusterpark Ltd. 15. / 52

16 3.3 Installation and initial configuration of individual golloader on other Linux hosts Installation of golloader Linux agent on other hosts can be performed only after initial configuration of GOL. For installation of golloader on other hosts the GOL full installation package must be used, just omit the unnecessary parts as goldb and GOL-GUI. For installation of golloader follow the instructions described in chapter 3.1 Preparing for GOL installation or start it from storage (HDD, flash disk) where it was stored after initial download. During installation the questions about goldb and GOL-GUI installation must be answered with n, because these components already are installed. In next step on question Would you like to install gol-loader package? type yes and press Enter. After checking of necessary components, a list of golloader software setup and installation size will be displayed to user. To continue installation type y and press Enter ; To cancel golloader installation type n and press Enter. Clusterpark Ltd. 16. / 52

17 After installation of loader its initial configuration is needed. It is required to use already installed golloader settings. Description of settings is available in Table 4. Table 4. Description of golloader installation settings. Settings field Clusterpoint Server Host? Clusterpoint Server Port? Storage name Username Password Loader name () Loader description() Description of the activities Type in goldb server address and press Enter. Specifies the network port through which golloader send data. By default the port - (5550) is used. Confirm with Enter. Specifies the name of database. By default the goldb. Type in the username for connection to goldb. By default the root or enter the username that was specified in the GOL configuration. Type in the password for connection to goldb. By default the - y2sp or enter the password that was specified in the GOL configuration. Type the name of server or computer on which the loader will be installed. This name will be used to identify host in goldb and GOL-GUI. Confirm with Enter. Field is not mandatory. In this field description of host can be written. Confirm with Enter. Now golloader agent installation has been completed and further configuration of settings can be carried out in GOL-GUI configuration section, as described in chapter 5 Linux golloader configuration. 3.4 Additional configurations for CentOS operating system If Linux CentOS operating system has been used it is recommended to perform following additional operations. Following commands should be entered in terminal mode: Add firewall rules. Before input of firewall rules it is recommended to become familiar with the iptables program editing commands. Clusterpark Ltd. 17. / 52

18 Execute the command: vi /etc/sysconfig/iptables Add following lines to the firewall settings table: -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT -AINPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport j ACCEPT After saving of table execute the command: /etc/init.d/iptablesrestart Add to the httpd process rights for writing in log folder. Execute following command: chcon -v --type=httpd_sys_content_t /usr/local/gol-gui/apps/logs Allow the httpd process the usage of network connections. Execute the command: setsebool -P httpd_can_network_connect 1 If Apache does not start at Linux boot time, than run levels must be installed by executing commands: chkconfig --listhttpd chkconfig --level 2 httpdon chkconfig --level 3 httpdon chkconfig --level 4 httpdon Clusterpark Ltd. 18. / 52

19 chkconfig --level 5 httpdon 3.5 Configuration of SNMP trap GOL has capability to download the network equipment's internal log entries using SNMP trap. To activate this capability some additional configuration operations must be performed. All operations can be performed in Linux terminal mode with root user rights: Open the file "/etc/apt/sources.lst" in editor. Append to the end of data source IP address line the "contrib non-free". If this file is not found, than this step can be ignored; If SNMP process is not found, it must be installed by executing following commands: apt-get update apt-get install snmp apt-get install snmpd apt-get install snmp-mibs-downloader download mibs For SNMP installation other commands and resources can be used according the current Linux version. Open the file "/usr/share/snmp/snmp.conf" in editor. Add new line "mibs +ALL" and save the file. The location of the file snmp.conf may differ from version to version; Settings in file "/usr/share/snmp/snmptrapd.conf" should be configured. Additional information can be found in but additional information about SNMP v3 can be found in ons. snmpd file location may differ from version to version: # traphandle all to GOL traps handler script. Path is just for example. #You should specify one where you will save the actual script traphandle default /0admin/snptrap/traphandle.02.sh.php # this disables traps logging to syslog donotlogtraps 1 # configure traps authentification # SNMP v1,v2 example authcommunity log,execute,net nvpublic # SNMP v3: TRAP user #createuser -e ENGINEID myuser SHA "my_auth_pass" AES "my_encryption_pass" createuser -e 0x v3trapuser SHA v3password AES v3coding # SNMP v3: INFORM user createuser v3trapuser SHA v3password AES v3coding # SNMP v3: Authorizing your user to do things with the received notifications authuser log,execute,net v3trapuser # SNMP v3: If you want to receive v3 traps (or informs) sent with noauthnopriv, authuser log,execute,net myuser noauth Execute the command: Clusterpark Ltd. 19. / 52

20 start snmptrapd Check the configuration by executing following commands: snmptrapd -v 3 -n "" -a SHA -A v3password -x AES -X v3coding -l authpriv -u v3trapuser -e 0x localhost 0 linkup.0 snmptrapd -C -v 3 -a SHA -A v3password -x AES -X v3coding -l authpriv -u v3trapuser localhost 0 linkup.0 Create PHP command file for SNMP process. For this the graphical or PHP command line editor can be used: <?php // The file we will be writing to. It's best to specify a full path, to avoid // any confusion over the current working directory $outfile = '/data.disk/trap/snmp_trap_'. date('ymd'). '.log'; // First, we'll open our outfile... if (!$ofp = fopen($outfile, 'a')) exit("oh No! I couldn't open my out file!"); //...and write the current timestamp to it, so we are certain the script was // invoked. You can remove this if you want, once you are sure it is working fwrite($ofp, date('y/m/d H:i:s')."\t"); // Next we'll write all the trap data to the file // We could use stream_copy_to_stream() for this, but I don't know if it is // available on your server so I won't do it here //fwrite($ofp,"data:\t"); while (!feof(stdin)) { $string = trim(fread(stdin, 1024)); $string = str_replace("\n", "\t", $string); $string = str_replace("\r", "", $string); fwrite($ofp, $string."\t"); } fwrite($ofp,"\n"); // We don't actually need a closing PHP tag and in many cases it is better // to omit it, to avoid unexpected whitespace being output. Save created PHP script in the folder which was specified in snmptrapd.conf faile. For example /0admin/snptrap/traphandle.02.sh.php ; If the snmpd and snmptrapd processes are not started during boot, than they must be added to boot manually. To perform that, execute following commands in terminal mode: Chkconfig snmpd on Chkconfig snmptrapd on Now SNMP trap is ready and golloader can be configured in GOL-GUI to start transfer data from a SNMP trap to goldb. Clusterpark Ltd. 20. / 52

21 4 GOL configuration after installation 4.1 Detection of goldb IP address If IP address is static and well known than this step can be omitted. Before proceeding with further GOL configuration it is necessary to determine the IP address assigned to the host on which GOL is installed. This can be done by executing the command ifconfig in Linux terminal mode: goldb IP address is specified in eth1 configuration. In example picture above it is seen as: inet addr: REMARK! This address must not be changed. Otherwise golloader agents will not be able to send log data to goldb and their reconfiguration should be done manually. Clusterpark Ltd. 21. / 52

22 4.2. Getting started with GOL To get started with GOL for further configuration in internet browser command line: On same server or computer where GOL is installed enter localhost/gol/ ; On any other computer in network enter [IP] indicates the server address on which GOL is installed. GOL-GUI access form opens (Picture 1). Picture 1. GOL access form. To start GOL configuration it should be accessed with Administrator rights. GOL installation already had set up default Administrator login and password: Username: admin Password: password The username should be written in the field with icon, but password in the field with icon. Press LOGIN button. The GOL-GUI work screen opens and the user can start working according to the rights granted. The access rights with which the user has accessed the GOL are shown on menu line - Logged in:. For example, user with administration rights will be displayed as admin: 4.3 Finishing the work with GOL To finish and log out from GOL-GUI at the end of menu line icon could be pressed. GOL-GUI access form opens again and tab in internet browser can be closed. Clusterpark Ltd. 22. / 52

23 5 Linux golloader configuration This chapter describes the sequence for Linux operating systems golloader agent configuration. 5.1 Initialization of new golloader agent In order to connect new golloader agent to GOL in window with GOL configuration elements (Picture 2) choose a category Hosts. Picture 2. Window with GOL configuration elements. The windows for list with installed agents will be opened; move to New hosts tab. Example is shown in Picture 3. Picture 3. Example with new GOL golloader agents. For further configuration of golloader on the host click icon. Window for configuration of golloader general settings will be opened (Picture 4). Clusterpark Ltd. 23. / 52

24 Picture 4. Window for configuration of golloader main settings. Configuration settings will be offered by default, but they can be changed if necessary. Description of changeable settings fields: o o o Poolcount: the amount of log entries which golloader is sending to goldb during each session can be specified. By default maximum 200 entries will be sent; Sleeptime (s): interval in seconds between each session can be specified. By default interval is set to 20 seconds; In order to activate agent and start log data sending process press on switch (disabled by default). Switch icon will be changed to indicating that log data sending is enabled; To save the configuration settings button Save must be pressed. Now golloader agent is turned on and is ready for further configuration. After entering of main settings the window for configuration of log data sources will be opened (Picture 5); Picture 5. Window for list of log data sources. Clusterpark Ltd. 24. / 52

25 5.2 Configuration of log data sources In order to add new log data source press icon source settings will be opened (Picture 6).. The window for configuration of log data Picture 6. Log data source configuration window. Description of log data source configuration settings is shown in Table 5, but examples with Linux log data sources are shown in Table 6. Table 5. Description of log data source settings. Field name Description Source name A short name of log data source must be written. Description The description of data source can be written. Not mandatory, but recommended. File directory Filename Type Storage Load previous logs The full path to the folder from which log data will be collected must be indicated. In this field the unchanging part of log file name must be written. For example if the files yum1, yum2, yum3, etc. (which are the rotation files), do exist than simply write yum. Corresponding log file type must be select from the dropdown menu. Specify to which of goldb log data storages the entries will be sent. By default it is main. Switch is indicating should the agent collect and send to the database historical entries from rotation files. By default position is Enable load this source files - sending of historical data is enabled. Can be moved to position - historical data won t be sent. Main switch for initialization of log entry sending process. By default switch is in position - log data sending will be started Clusterpark Ltd. 25. / 52

26 immediately after saving of configuration. It can be moved to position, if data sending directly after saving of configuration is not necessary. It can be turned on later though. Before the connection of log data source it is necessary to check their names and location. Table 6. Examples for Linux log data sources. Log name File directory File name Type YellowdogUpdater, Modified /var/log/ yum Load Custom Cron /var/log/ cron Syslog Linux security /var/log/ secure Syslog System messages /var/log/ messages Syslog Apache access /var/log/httpd access_log Apache2 access Apache error /var/log/httpd Error_log Apache2 error Yum Cron Secure Messages Apache access Apache error Log file keeps the name and work time of installed programs and packages. Log file keeps entries of executed scheduled software installation activities. Log file keeps notifications about user login and logout actions. Log file keeps overall performance of the system. Log file keeps access to the Apache server data. Log file keeps Apache server error notifications. Usually Linux log file names are left as they are after installation, but sometimes users are configuring the names for rotation files to identify them with date in file name. In these cases file names must be specified as regular expressions. Examples how to create file names as regular expressions are shown in Table 7. Table 7. Examples of log file names as regular expressions. Log types recognized by default Messages, syslog cps apache2err apache2acc snmptrap_exp If file name is not modified, the name to write in field File name is specified in left column. /^messages(-[0-9]{8,8})?$/ /^{cpart}([.]log){0,1}([.][0-9]{1,2}){0,1}(([.]gz){0,1} ([.]bz2){0,1})$/ /^{cpart}(_[0-9]{2,2}){3,3}[.]log$/ /^{cpart}(([-._]\d{1,8})(([.]gz){0,1} ([.]bz2){0,1})){0,1}$/ /^{cpart}(([-._]\d{1,8})(([.]gz){0,1} ([.]bz2){0,1})){0,1}$/ /^{cpart}_[0-9]{8,8}[.]log(([.]gz){0,1} ([.]bz2){0,1})$/ Small training how to create regular expressions can be found at - The above described configuration steps should be performed for each log file that is meant to store in database. The statuses of configured log data files can be checked in tab Sources. Example is shown in Picture 7. Clusterpark Ltd. 26. / 52

27 Picture 7. List of configured log data sources. Icon is indicating that golloader agent is active and sending the log entries from corresponding log files to database. Icon is indicating that log entries from corresponding log file are not sent. Possible reasons may be: The log file name or folder has been entered incorrectly; Specified log file do not have entries yet. When all necessary log files are configured, with link Back to host list return to the list of configured hosts (Picture 8). Picture 8. List with configured hosts. Clusterpark Ltd. 27. / 52

28 6 Installation and configuration of Windows golloader agent (WinGol) Installation package can be downloaded to hard disk from the internet address provided by developer. From there it can be copied to other media for installation on Windows servers and computers (hosts). 6.1 Preparing for WinGol installation Prior installation the goldb IP address, database login name and password must be identified. In order to do that: Start GOL-GUI and login with administrators rights; In main menu open Configuration; In the window with configuration elements chose General; Open the tab Storages; Settings necessary for installation are specified in section golcfg oposit the names: o Address: goldb IP address; o Username: login name for connection to the database, by default - root ; o Password: password for connection to the database, by default - y2sp. 6.2 Installation of WinGol After download of installation package open the folder where it is located and run the installation package. WinGol installation starting window will be opened (Picture 9). Picture 9. WinGol installation starting window. To continue installation press on button Next. In order to cancel the installation press Cancel button. By pressing on Next the window (Picture 10) where name of the host and its description must be specified will be opened. Clusterpark Ltd. 28. / 52

29 Picture 10. Window for input of host name and description. Description of input parameters is shown in Table 8. Table 8. Description of input parameters for host name. Name of field Host Name: Host Description: Description Input the name of the host. It will be used for identification of host in GOL-GUI visualization tools. Host name is determined automatically from Windows settings. It can be changed if necessary. Input hosts broader description. After input of information press Next to continue installation. The window for input of goldb server IP address, login and password (Picture 11) will be opened. Picture 11. Window for input of goldb connection settings. Clusterpark Ltd. 29. / 52

30 Description of input settings for connection to goldb is available in Table 9. Table 9. Description of goldb connection settings. Name of field Description Server IP: Input IP address where goldb is located. Username: Input username for connection to goldb. Password: Input password. The username and password can be can be found in main menu under Configuration / General / Storage corresponding fields Username and Password. After input of information press button Next or button Back, if previously entered information should be changed. After pressing on Next window for input of WinGol installation folder will be opened (Picture 12). Folder can be specified from the list of existing folders by pressing the button Browse. Default folder name C:\WinGol is already specified in input field. Picture 12. Window for input of WinGol installation folder. After input of installation folder press button Next. If the new WinGol version is installed over previous one, than additional dialog, informing that installation folder already exist, will appear (Picture 13): Picture 13. Dialog for approval of WinGol installation folder. Clusterpark Ltd. 30. / 52

31 To accept installation in folder that already exist press button Yes. If WinGol should be installed in other folder press No and dialog will return to window for input of installation folder (Picture 12). By pressing Yes the window for input of folder where log data prepared for sending to goldb will be collected will be opened (Picture 14). Folder can be selected from the list of existing folders by pressing button Browse. Default name of the folder C:\WinGol already is specified in input field. Picture 14. Window for input of folder where log data will be collected. After input of folder press button Next. In the next window (Picture 15) the folders selected for installation will be shown. If the names of folders or other settings should be changed press button Back. To continue with installation press button Install. Picture 15. Window with overview of installation folders. Clusterpark Ltd. 31. / 52

32 After successful installation the window with overview of WinGol installation status will appear (Picture 16). Picture 16. Installation status window. To close it pres Finish. All WinGol processes necessary for collection and transmission of log data will be started automatically. Information about connected host will appear in GOL-GUI approximately after 10 sec. and will be available for further configuration. 6.3 WinGol configuration in GOL To finish configuration of WinGol, run GOL-GUI in the browser and login with administrator rights: In main menu open Configuration; In the window with configuration elements chose Hosts; Open tab New hosts (Picture 17) with list of newly installed loaders; Picture 17. Windows with list of newly installed loaders. For further configuration of loader press on icon. The window for configuration of WinGol general settings will be opened (Picture 18). Clusterpark Ltd. 32. / 52

33 Picture 18. Window for configuration of WinGol main settings. Description of loader configuration input settings is shown in Table 10. Table 10. Description of loader configuration settings. Name of the field Name Description Pool count Sleep time (s) Switch Description Specifies the name of the host where WinGol is installed. Broader description of the host. The amount of log entries which WinGol is sending to goldb during each session can be specified. By default maximum 200 rows will be sent. Interval in seconds between each session can be specified. By default interval is set to 20 seconds. In order to activate agent and start log data sending process press on switch (disabled by default). Switch icon will be changed to data sending is enabled. indicating that log Settings must be saved by pressing on button Save. Now the WinGol agent is enabled and ready for further configuration. The window for list of log data sources will be opened. To configure the log data sources for particular loader press New sources tab (Picture 19) where all automatically recognized MS Windows event types will be listed. REMARK! If the New Sources tab in configuration window has index (0) the GOL- GUI link in the browser should be refreshed. Clusterpark Ltd. 33. / 52

34 Picture 19. List of automatically recognized MS Windows events. The name of event is specified in column Name. The in the column Enable does mean that corresponding log data transmission is not started and it must be enabled. That can be done by pressing on icon. The window for log data transmission settings configuration will be opened (Picture 20); In the settings fields information received from loader has been already shown. It is not recommended to change them, otherwise the loader won t work correctly and log data won t be sent to goldb. Parameters that can be changed at this step are described in Clusterpark Ltd. 34. / 52

35 Table 11. Picture 20. Log data transmission configuration window. Clusterpark Ltd. 35. / 52

36 Table 11. Description of log data transmission settings. Name of the field Source name Description Load previous logs Enable load this source file Description Specify the name of event according the MS Windows classification. Not recommended to change. Description of event. Switch in position inform, that historical events prior current time and date will be sent to the goldb as well. If that is not necessary, press on it and switch will be moved to position off -. To start transmission of the data switch must be pressed (disabled by default). Switch icon will be changed to indicating that data transmission is enabled. Configuration settings must be saved by pressing button Save. The above steps can be performed to other log data sources which are intended to be transmitted to database as well. Status of enabled events can be checked in the tab Sources (Picture 21). Icon indicates that loader is transmitting the entries from corresponding data file to the goldb; Picture 21. List with enabled for transmission data sources. When all events are enabled with link Back to host list return to the list with configured hosts (Picture 22). Picture 22. List with configured hosts. Clusterpark Ltd. 36. / 52

37 Switch at the position in column Enable indicates that the loader on particular host is enabled, but icon in front of host name indicates that the loader is transmitting data to database. Clusterpark Ltd. 37. / 52

38 7 WinGol and Linux golloader uninstallation 7.1 Preparing GOL prior WinGol uninstallation Before WinGol uninstallation from the host it must be disconnected from goldb: Run GOL-GUI in web browser and login with administrators rights; Open Configuration in main menu; In the window with configuration elements chose Hosts, window with list of all registered in GOL devices will be opened (Picture 23). Picture 23. List of devices registered in GOL. In order to disable the log data transmission press the switch in column Enable, switch will be changed to indicating that particular device is disconnected from goldb. Accordingly, the icon opposite the host name in the column Name from will be changed to as additional indication that particular device is disconnected (Picture 24). Picture 24. Example of list with disconnected device. To remove the device from the list and from network chart it must be deleted by pressing in the column Remove (Picture 24). The dialog window for approval of removing will be opened (Picture 25). Clusterpark Ltd. 38. / 52

39 Picture 25. Dialog window for approval of device removing. By pressing OK button information about device will be moved to the tab Removed hosts (Picture 27). Picture 26 shows the list of hosts without deleted entry. Picture 26. List of hosts without deleted device. Removed hosts index (1) indicates the number of records currently in tab (Picture 27). Picture 27. List with deleted entry. The inscription in Enable column indicates that the host has been successfully removed from GOL and MS Windows log records are no longer sent. In the next step find the appropriate folder with installed WinGol on the host (default folder - C: \ WinGol) and run the program unins000.exe. WinGol uninstallation window will be opened (Picture 28). In order to start WinGol uninstallation process press button Yes. By pressing on button No uninstallation will be terminated; Picture 28. WinGol uninstallation dialog. Clusterpark Ltd. 39. / 52

40 The program will automatically stop all processes and remove WinGol and its components from hard disk; After successful WinGol uninstallation the window with appropriate message will be displayed (Picture 29). Picture 29. Window with message about successful WinGol uninstallation. To close the message window press button OK. After uninstallation the WinGol folders will remain on hard drive. They must be deleted manually if they won't be needed anymore. 7.2 Linux golloader and GOL uninstallation In order to uninstall golloader and GOL appropriate to Linux version commands must be executed: Debian 1) Switch to user root su l: 2) Uninstall golloader packages and remove folders: dpkg -P gol-loader && rm -rf /usr/local/gol-loader; 3) Uninstall GOL packages and remove folders: dpkg -P gol-gui && rm -rf /usr/local/gol-gui; 4) Uninstall goldb packages and remove folders: Fedora dpkg -P cps2-server && rm -rf /usr/local/cps2 1) Switch to user root su-l: 2) Uninstall golloader packages and remove folders: Clusterpark Ltd. 40. / 52

41 yum remove gol-loader.noarch && rm -rf /usr/local/gol-loader 3) Uninstall GOL packages and remove folders: yum remove gol-gui.noarch && rm -rf /usr/local/gol-gui 4) Uninstall goldb packages and remove folders: yum remove cps2-server.x86_64&& rm -rf /usr/local/cps2 CentOS 1) Switch to user root su-l: 2) Uninstall golloader packages and remove folders: yum remove gol-loader* && rm -rf /usr/local/gol-loader 3) Uninstall GOL packages and remove folders: yum remove gol-gui* && rm -rf /usr/local/gol-gui 4) Uninstall goldb packages and remove folders: yum remove cps2* && rm -rf /usr/local/cps2 Clusterpark Ltd. 41. / 52

42 8 Restore WinGol installation This chapter describes the installation sequence for WinGol recovery after uninstalling from the computer or server. Restoration of WinGol installation may be necessary if the host that has already been registered in GOL, is again necessary to connect to GOL for log data transmission. REMARK! If WinGol on the host was uninstalled than at first WinGol installation must be carried out as described in the chapter6.2 Installation of WinGol. It must be noted that the Host Name must be the same as it is registered in GOL Configuration section Hosts tab Removed hosts. After the host connection to the network it must be enabled in GOL: Run GOL-GUI in web browser and login with administrators rights; Open Configuration in main menu; In the window with configuration elements chose Hosts, window with list of all registered in GOL devices will be opened; In system configuration section open tab Removed hosts (Picture 30). Picture 30. List with removed hosts. Press on icon in the column Edit, in line with appropriate host; Host configuration window will be opened (Picture 31); press on the switch in the line with host name. Switch will be changed to is enabled for sending log data; which means that host again Picture 31. Host configuration window. Clusterpark Ltd. 42. / 52

43 If WinGol on host is installed again, it is necessary to specify in GOL configuration the agent's name: In the Configure host window (Picture 31) press on icon at the end of line with host name. Host connection configuration window will be displayed (Picture 32). Picture 32. Host connection configuration window. In the dropdown menu Link to a new loader is a record New loaders which means that new loader is available; Open Link to a new loader dropdown menu (Picture 33) and from the list choose the appropriate loader. Picture 33. Drop down list with available loaders. The button Save (Picture 32) must be pressed for confirmation of configuration. Now the host connection to the GOL has been restored. Clusterpark Ltd. 43. / 52

44 9 goldb management environment goldb has been created using the Clusterpoint XML document oriented NoSQl database engine. To help manage goldb servers user-friendly database management environment is available. 9.1 Access to the goldb user's environment To get to the goldb user's environment a web browser program can be used. In to the browser address field: on the same host where GOL was installed enter "localhost:5580"; for access from another host enter [IP] indicates the host address on which GOL is installed. goldb terminal user access form will be opened (Picture 34). Picture 34. goldb terminal user access form. Default administrator's name and password already has been set by goldb installation: User name: root Password: password When the corresponding access parameters are entered press the button LOGIN. 9.2 Overview of the goldb server resources After authentication in goldb, user environment initially displays information about server resources in use (Picture 35). Picture 35. Overview of goldb server resources. Clusterpark Ltd. 44. / 52

45 Overview shows information about server CPU load, hard disk and memory utilization and the current version of the database. Attention should be paid to the amount of memory used; its availability is a very important for goldb to work productively. This information can be used for planning of the necessary resources. 9.3 Database management In the menu items Single Storages and Cluster Storages management of individual databases and creation of new instances is available. These sections features detailed information about each database memory and hard disk utilization, amount of documents (log entries) consolidated, amount of indexed words and database status (Picture 36). Picture 36. Database information window Stop and activation of individual databases In cases when stop or activation of the database is needed: a) In order to stop all databases press the button ; b) In order to activate all databases press button ; c) To stop or activate individual database press button and chose the items Start or Stop accordingly Deleting documents from database Deleting of entries from database, with keeping of data sources, can be executed in following order: a) Stop the golloader agents as described in the chapter 7.1 Preparing GOL prior WinGol uninstallation; b) Open the Cluster Storages item in goldb terminal; c) In the window that opens press on goldb in the column Cluster storages; Clusterpark Ltd. 45. / 52

46 d) Chose and press the link Run command in the tab More: e) New window will be opened where from dropdown chose command clear: f) In order to accept command and delete the entries in database press button in the right edge Commands that can be executed in database For goldb management several built-in commands can be used. How to access them is described in chapter Deleting documents from database starting from step b). Description of commands is available in Table 12. Table 12. Commands available for goldb management. Command Search Retrieve Lookup List-last Clear Delete Status Reindex Rebalance Description Command can be used for search of words and phrases in the database and displaying of the results. Command for finding and retrieval of document with displaying of full content by its ID nr. Command for finding of the document by its ID nr. View the last entries in the database. Delete all the database entries Delete database Command for viewing of detailed database status Command for re-indexing of the data Command can be used for rebalancing of the goldb content among all nodes These and other advanced commands can be executed in the menu item Tools. In example (Picture 37) with a red underline is marked the replaced value in goldb database for finding the word "error". Clusterpark Ltd. 46. / 52

47 9.6 goldb users and passwords Picture 37. Example with changed values in goldb. In order to create or edit goldb users press the menu item Administration -. The list with current users will be opened (Picture 38). By default in GOL are to users: 1) User root with password password is used for goldb administration; 2) User gol with password y2sp provides golloader authentication to the server for sending of log entries to the database. Picture 38. List with goldb users. In order to change the password check the box ( ) and press Edit User. New window for changing of the password will be opened; Clusterpark Ltd. 47. / 52

48 New password must be entered in to the both Password fields and saved by pressing button Save (Picture 39). Picture 39. Window for change of goldb user password. Other settings can be edited in this window as well: User name can be changed by writing new one in the field Login name; Full name of the user can be specified in the field Full name; From items in the drop down menu User Type the user rights can be chosen; User can be enabled or disabled by adding or removing the check in the Enabled field box ( ). 9.7 Database license and its extension If the database license term is over, the license may be renewed upon receipt of a new license file. After receiving of the file it must be uploaded to the database: 1) In the menu item Administration press the arrow on the button and from the dropdown menu chose item License; 2) Windows for license installation will be opened. Under Install new license check the radio button with appropriate method for license renewal: a) In order to upload the file check the item Upload license file and press the button Browse ; b) In order to renew the license with content from the license file check the item Copy license content file in the empty field. and copy the content of the Clusterpark Ltd. 48. / 52