Release 3.0 Insights Storage Operation and Maintenance Guide. Revision: 1.0

Transcription

1 Release 3.0 Insights Storage Operation and Maintenance Guide Revision: 1.0

2 Insights Storage Operation and Maintenance Guide Portions of the documents can be copied and pasted to your electronic mail or word-processing applications for your personal use only, but cannot be distributed to third parties. In no event may you copy or use this information for any commercial purposes except the operation of products from Procera Networks, Inc. and you may not transmit this information to third parties without the consent of Procera Networks, Inc. IT IS ILLEGAL TO COPY (FOR OTHER THAN BACK-UP PURPOSES) THE CONTENTS OF THIS DOCUMENTATION OR TO POST THE CONTENTS ON THE INTERNET WITHOUT THE EXPRESS PRIOR WRITTEN CONSENT FROM AN AUTHORIZED OFFICER OF PROCERA NETWORKS, INC. OR PROCERA NETWORKS AB. THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. PacketLogic and the PacketLogic logo are registered trademarks of Procera Networks AB in Sweden and certain other countries. Procera Networks and the Procera Networks logo are registered trademarks of Procera Networks, Inc. All other trademarks mentioned in this document are the property of their respective owners. Copyright Procera Networks ii

3 Table of Contents 1. Introduction Prerequisites New in this release Nodes can be added or replaced in Insights Storage cluster Vertica upgraded to version Other changes Upgrade Insights Storage Upgrade procedure Product overview Insights components Communication between components The Insights Storage cluster Architecture of the cluster Failure recovery K-safety Maintenance TUI options S) Status C) Cluster Management C) Create New Cluster A) Add Host to Cluster R) Remove Host from Cluster I) Initialize Cluster A) Add Node(s) P) Replace Node P) Configure Packages M) Monitoring S) Show SNMP Configuration E) Enable SNMP D) Disable SNMP T) Test SNMP D) Database Management N) Stop This Node D) Stop Database R) Refresh S) Start Database Common procedures Replacing a node Shutting down nodes Moving the PLSD process from PRE to PIC iii

4 iv

5 1. Introduction This document is intended as a guide to operate and maintain Insights Storage 3.0. Insights Storage 3.0 stores data extracted from traffic running through a PRE and pushed to Insights Storage from systems (PRE or PIC) running PLSD. Engineering Insights provides a web-based frontend for viewing the data stored in Insights Storage Prerequisites Version 3.0 of Insights Storage can be installed on PL1650 systems using firmware version or later of released Packet- Logic firmware versions. See the Insights Storage Installation Guide for instructions on installing Insights Storage. The PLSD systems providing Insights Storage 3.0 with data must be of firmware version 17.1 or later. 1

6 2

7 2. New in this release This chapter highlights the main changes since release 2.4 of Insights Nodes can be added or replaced in Insights Storage cluster Insights Storage now supports adding or removing nodes after the Vertica cluster has been configured and initialized. There are two new options for adding and replacing nodes in the TUI under the Cluster Management menu, see Section 5.1.2, C) Cluster Management. Note Once a cluster has been initialized, the IP address of a node in the cluster cannot be changed. A workaround is to remove the node from the cluster and add it back again with a different IP address, see Section 5.2.1, Replacing a node Vertica upgraded to version 8.1 Insights Storage now runs Vertica version 8.1. Read more in the Vertica 8.1 Release Notes Other changes Renaming of TUI menu items for Insights Storage. The Configure Cluster menu is now called Cluster Management and the Cluster Storage menu is called Database Management. See Section 5.1, TUI options. 3

8 4

9 3. Upgrade Insights Storage Caution When upgrading the Insights Storage PSF package, it is possible to remove all Insights Storage traffic and score data. To avoid data loss, ensure that N is selected when the question Perform a full uninstall of the old version before installing the new? is prompted during the upgrade Upgrade procedure Insights Storage 3.0 can be upgraded from version 2.3 or later. In a multi-node cluster, the following upgrade procedure is performed on all nodes. Note The nodes in a multi-node cluster must be upgraded one at a time. The I) Install/Update modules command must be completed on one node before the upgrade on the next node is started. Procedure 3.1. TUI upgrade procedure 1. Connect to the Insights Storage PIC system using PacketLogic Client. In the File Manager, select the Upgrade files directory and upload the insights-storage PSF package using the Upload file button. 2. In the TUI of the Insights Storage system, select M) Service management > C) Customisations. 3. Select I) Install/Update modules. Customisations ========================================== I) Install/Update modules U) Uninstall modules L) List installed modules Back) Go back one level Main) Go back to top level Index) Index Quit) Quit Help) Help [172.xx.xx.xx] > I 4. Select the the Insights Storage module. Caution Ensure that N is selected when the question Perform a full uninstall of the old version before installing the new? is prompted. Selecting y will delete all Insights Storage traffic and score data, along with configuration information. Select module to install: 1) insights-storage-v3.0 A) Abort 5

10 Upgrade Insights Storage > 1 insights-storage is already installed on the system. The upgrade will cause all module services to be shut down and restarted. Install new version? (y/n): y Perform a full uninstall of the old version before installing the new? (y/n): N ok: down: /var/service/psf-insights-storage: 1s, normally up ok: down: /psf/insights-storage/services/cron: 0s ok: down: /psf/insights-storage/services/pldcd: 0s ok: down: /psf/insights-storage/services/pliid: 1s Stopping Insights Storage Detected an upgrade, not stopping the database ok: run: syslog-ng: (pid 80822) 1s ok: run: pldbd: (pid 80994) 0s Database is running, starting migration... psf: Enabled PSF module insights-storage ok: run: syslog-ng: (pid 85729) 1s ok: run: pldbd: (pid 85903) 0s Customisation Module Package 'insights-storage' installed. 6

11 4. Product overview This chapter contains an overview of Insights Storage and linked systems Insights components An Insights system consists of the following components: One or more PRE systems with network traffic passing through One or more PacketLogic Statistics Daemons (PLSD) (configured on the PRE or on PIC systems) that accumulate data extracted from the network traffic An Insights Storage cluster that stores the data An Engineering Insights application that visualizes the stored data Communication between components The engine in PRE sends connection updates to PLSD. The data is accumulated and pushed to Insights Storage, where the information is organized and stored in database tables. Engineering Insights requests data from Insights Storage, which is presented and visualized in the Engineering Insights graphical user interface The Insights Storage cluster Insights Storage consists of a single node or a cluster of 3 or more nodes holding a distributed database. For high availability, a minimum of three nodes is recommended. Each node runs an instance of HP Vertica. Refer to the HP Vertica documentation for detailed information about the Insights Storage database architecture and functions Architecture of the cluster The architecture of a cluster is designed to distribute physical storage of data, and to allow parallel query execution over a potentially large collection of resources. 7

12 Product overview The following concepts are used: Host A system with a processor, RAM, hard disk and network interface. Hosts do not share disk space or memory with each other. Instance A running HP Vertica process and disk storage on a host. Only one instance of HP Vertica can be running on a host at any time. Node A host configured to run an instance of HP Vertica. A node is a member of the database cluster. For a database to have the ability to recover from the failure of a node, at least three nodes are required. Cluster A collection of nodes that is bound to a database. Database A cluster of nodes that, when active, can perform distributed data storage and SQL statement execution Failure recovery The failure recovery function in Insights Storage is based on the distributed nature of the database. An Insights Storage database that consists of a cluster of 3 or more nodes is K-safe (see Section , K-safety ), which means that one node can fail at any time without causing the database to shut down. When a failed node comes back online and rejoins the database, it recovers its lost data by querying the other nodes K-safety K-safety is a measure of fault tolerance in the database cluster. The value K represents the number of additional replicas of the data in the database (for example, K-safety=1 means that there are 2 copies of the data). These replicas allow other nodes to take over for failed nodes, allowing the database to continue running while still ensuring data integrity. If more than K nodes in the database fail, some of the data in the database may become unavailable. In that case, the database is considered unsafe and automatically shuts down. The value of K can be 0, 1 or 2. If a database that has a K-safety of 1 loses a node, the database continues to run normally. Potentially, the database could continue running if additional nodes fail, as long as at least one other node in the cluster has a copy of the failed node's data. 8

13 5. Maintenance This chapter contains information about Insights Storage maintenance TUI options The Text-based User Interface (TUI) is available when connecting to the console of the Insights Storage system with SSH to the Admin IP address port 42002, or through a serial connection. The following Insights Storage TUI options are available after installation of the Insights Storage PSF package, under M) Service management > C) Customisations > S) Insights Storage. For installation instructions, see the Insights Storage Installation Guide S) Status The Status menu displays the following information about the status of the Insights Storage system: Cluster Configured This item shows information about the database cluster. If marked YES, the cluster is configured and available. Cluster Initialized If marked YES, the database has been created. Database Running If marked YES, the database is available for storage. If marked NO, the database can be started using D) Database Management > S) Start Database in the Insights Storage menu. The installed PSF package, its version and the nodes it is installed on are also listed in the Status display C) Cluster Management The Cluster Management menu displays configured hosts and contains options to add nodes to or remove nodes from the cluster. The first four menu items are only available before a cluster has been configured and initialized, and the last two will be available after the cluster has been initialized C) Create New Cluster This option is used to create the cluster and add the IPv4 address of the local node to it. This option is only available before nodes have been added to the cluster. Note Do not use or localhost. It will not be possible to add the node to a cluster if or localhost is used A) Add Host to Cluster This option is used to add the IPv4 address of a host that will be a part of the cluster R) Remove Host from Cluster This option is used to remove a host from the cluster I) Initialize Cluster This option creates the database. 9

14 Maintenance Note Once the cluster has been initialized, the IP address of a node in the cluster cannot be changed. A workaround is to remove the node from the cluster and add it back again with a different IP address, see Section 5.2.1, Replacing a node A) Add Node(s) This option is used to add a node to the cluster after the cluster has been initialized. New nodes are added by entering a comma-separated list of IP addresses. Once the nodes are added to the cluster, a background task is run to re-balance the data. Note All nodes must have the insights-storage-psf of the same version installed before they are added to the cluster. Note This option is only available in multi-node clusters P) Replace Node This option is used to replace a node after the cluster has been initialized. It can be used to add an RMA-replaced unit back to the cluster with the same IP address as the faulty unit, or to replace a functioning node with another one with a different IP address. A node is replaced by first entering the IP address of the node to be removed and then the IP address of the node to be added. See Section 5.2.1, Replacing a node. Note All nodes must have the insights-storage-psf of the same version installed before they are added to the cluster P) Configure Packages The Configure Packages menu contains options to configure retention time for the traffic perspective data. Data is retained for 39 days by default, provided there is enough disk capacity. Selecting the traffic package in the Configure Packages menu will prompt for a configuration value to view or edit. To configure data retention time, select 1) Drop Partition Interval and then S) Set new value. This option is only available after a cluster has been configured and initialized. The traffic perspective has to be installed for this option to function M) Monitoring This menu contains options to configure and test Insights Storage SNMP traps S) Show SNMP Configuration This option will display IP address, port and community string of the configured SNMP monitors E) Enable SNMP This option will enable SNMP monitoring for Insights Storage. Note that PacketLogic SNMP monitoring is enabled separately (under M) Service management > S) SNMP configuration of the PacketLogic system) and does not affect Insights Storage SNMP monitoring. When selecting this option, IPv4 address, port and community string of the SNMP trap destination monitors are specified. 10

15 Maintenance D) Disable SNMP This option will disable SNMP monitoring for Insights Storage. Note that PacketLogic SNMP monitoring is disabled separately (under M) Service management > S) SNMP configuration of the PacketLogic system) and does not affect Insights Storage SNMP monitoring. This option is only available after Insights Storage SNMP has been enabled T) Test SNMP This option will send an SNMP trap test message to the configured monitors. This option is only available after Insights Storage SNMP has been enabled D) Database Management The Database Management menu contains options to manage the database. This menu is only available after a cluster has been configured and initialized N) Stop This Node This option will stop the service running on that specific node D) Stop Database This option will shut down the database. When the database is shut down, perspectives or Insights Storage may not be installed or uninstalled. This option will only be available when the database is running R) Refresh This option will reload the Database Management TUI menu S) Start Database This option will start the cluster and the database, and will only be available if the database is not running Common procedures Replacing a node A node can be replaced after the cluster has been initialized. If the node to be replaced is functioning, the new node can have a different IP address. A faulty node, as in the case of an RMA replacement, can be replaced by a node with the same IP address as the old one. For example, an RMA unit is received in replacement of a faulty unit in a three-node cluster. In this scenario, the replacement unit will be configured in the same way as the faulty one and have the same IP address. To replace the node, perform the following steps: 1. Remove the faulty unit from the network. 2. Install the new unit. 3. Install the insights-storage-psf of the same version on the new unit. (See the Insights Storage Installation Guide for instructions.) 11

16 Maintenance 4. On one of the healthy units in the cluster, go to the the Insights Storage menu of the TUI. 5. Enter C) Cluster Management > P) Replace Node. 6. Enter the IP address of the node to be removed and then the IP address of the node to be added. In this scenario, the same address is entered on both prompts Shutting down nodes When nodes in the Insights Storage cluster need to be shut down, rebooted, or if a firmware upgrade is performed, each node has to come back up properly before shutting down or upgrading any of the other nodes. If the database does not run after upgrading nodes (check the status item Database Running under the TUI option S) Status), it can be restarted using the options D) Database Management > S) Start Database Moving the PLSD process from PRE to PIC If the PLSD process needs to be moved from a PRE to a dedicated PIC for Insights data export to work properly, use the following process. 1. Add the IP address of the PRE using the TUI options N) Network configuration > S) Remote system. 2. Set INSIGHTS_TRAFFIC_ENABLED to False on the PRE. The value is configured in the Insights directory in the System Configuration Editor in the PacketLogic Client. Host Stats should still be enabled on the PRE. 3. Disable localstats in the PRE TUI using M) Service management > L) Local statistics. 4. Enable Insights and Host Stats in the System Configuration Editor in the PacketLogic Client on the PIC. INSIGHTS_TRAFFIC_ENABLED is configured in the Insights directory, and HOST_STATS_ENABLED is configured in the Host Stats directory. 5. The Insights and Host Stats system configuration values that specify NetObject or session context paths (previously configured on the PRE) need to be configured on the PIC. Any other non-default system configuration values that were previously configured on the PRE should now be configured on the PIC. 6. Remove the data collectors on the PRE using the TUI options M) Service management > I) Insights > C) Data collection > D) Delete storage node. 7. Add the data collectors on the PIC using the TUI options M) Service management > I) Insights > C) Data collection > A) Add storage node. 12