Plexxi HCN Plexxi Control High Availability Installation, Upgrade and Administration Guide Release 4.0.0

Transcription

1 Plexxi HCN Plexxi Control High Availability Installation, Upgrade and Administration Guide Release May 3, Innovative Way - Suite 3322 Nashua, NH Tel PLEX (7539)

2 Legal Notices The information contained herein is subject to change without notice. Plexxi, the Plexxi logo, and LightRail are registered trademarks, and Plexxi HCN, Plexxi Control and Plexxi Connect are trademarks of Plexxi, Inc. in the United States and other countries. Other product or service name may be trademarks or service marks of others. No part of this documentation may be reproduced in any form or by any means or used to make any derivative work (such as translation, transformation, or adaptation) without written permission from Plexxi, Inc. Plexxi, Inc. reserves all rights of copyright in this documentation. PLEXXI, INC. PROVIDES THIS DOCUMENTATION AS IS, WITHOUT WARRANTY, TERM, OR CONDITION OF ANY KIND, EITHER IMPLIED OR EXPRESSED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES, TERMS, OR CONDITIONS OF MERCHANTABILITY, SATISFACTORY QUALITY, NON-INFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE. Plexxi, Inc. reserves the right to make changes to equipment design or program components described in this documentation, as progress in engineering, manufacturing methods, or other circumstances may warrant. No responsibility is assumed for the use of Plexxi, Inc. software or hardware, all rights, obligations and remedies related to which are as set forth in the applicable sales and license agreements. Plexxi, Inc. 100 Innovative Way - Suite 3322 Nashua, NH Tel: PLEX (7539) Published May 3, 2018 Printed in United States of America. Copyright 2018 Plexxi, Inc. All rights reserved. The Plexxi Switch system is classified as a class 1 telecommunications laser product employing embedded class 1 lasers and complies with the following: THIS PRODUCT COMPLIES WITH FDA RULE 21 CFR SUBCHAPTER J IN EFFECT AT DATE OF MANUFACTURE. PRODUCT COMPLIES WITH 21 CFR AND PRODUIT CONFORME SELON LE SOUS CHAPITRE J DU DOCUMENT FDA RÈGLE 21 CFR EN VIGUEUR LORS DE LA DATE DE FABRICATION. PRODUIT CONFORME SELON 21CFR ET Electrotechnical Commission (IEC) , This product is classified as a: CLASS 1 LASER PRODUCT APPAREIL À LASER DE CLASSE 1 This unit is intended to be installed in a Restricted Access Location only with access only by trained personnel. Warning: The primary hazards of exposure to invisible laser radiation from an optical fiber communications system are: Damage to the eye by viewing an unterminated optical fiber or fiber optic connector. Damage to the eye from invisible laser radiation from viewing a cut fiber or a broken fiber. Never attempt to view optical connectors that may be emitting laser energy and always avoid possible exposure to invisible optical laser radiation. Using optical fiber scopes or magnifying lenses may increase the possibility for an eye hazard. It is recommended that you use an optical power meter to determine if there is optical laser radiation present or use a remote video display inspection tool to inspect connectors. 2 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

3 Table of Contents Legal Notices... 2 About this Document... 5 Related Documentation... 5 Contacting Plexxi Support About Plexxi Control High Availability... 6 HA Clustering Software... 7 Management Node for HA... 7 Plexxi Control HA... 7 Database Nodes for HA... 7 Virtual IP (VIP)... 7 Synchronization Installation Considerations... 8 HA System and Software Prerequisites... 8 Ansible Requirements... 8 Host Requirements... 8 Operating System... 8 Plexxi Control Requirements... 8 Host Requirements... 8 Supported VM Hypervisors for Plexxi Control... 8 Operating System... 8 Security Considerations... 9 Secure Passwords Download Files from Plexxi Install PCIC on the Management Host Create a Python Virtual Environment Install PCIC in the Virtual Environment Generate the SSH Key Create HA VMs and Install Linux CentOS on each VM Prepare the Plexxi Control Nodes Create a VM for Each Node Install Linux Changing the Linux Root User Password Selecting an Action Selecting the Time Zone Completing the Installation Set Up the Ansible Files Inventory file Inventory File - Single Instance Inventory File - High Availability Inventory File: HA and Ansible Playbook Variables Configure STONITH STONITH Parameters, Single Host Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

4 STONITH Parameters - multiple hosts Plexxi Control High Availability Operations Install Plexxi Control High Availability Upgrade Plexxi Control High Availability Update Plexxi Control High Availability Uninstall Plexxi Control High Availability Verifying High Availability Pacemaker Verifying PostgreSQL HA Verifying Plexxi Control High Availability Administering Plexxi Control Hosts Changing Default Linux Passwords on a New VM Changing the Plexxi Control Password Testing Remote Access to the Linux Host Optional Configurations Configuring Network Settings for a Device Configuring DNS Troubleshooting Menu Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

5 About this Document This document describes: Plexxi Control High Availability how to set up the Plexxi Control Installation and Configuration (PCIC) tool. This is a one-time procedure. how to create VMs for Plexxi Control in the VMware environment. how to install Linux on the VMs how to deploy Plexxi Control on the VMs or bare metal servers how to upgrade Plexxi Control how to manage High Availability Related Documentation The following additional documentation supports this release: Plexxi Compatibility Matrix, Version 10 or greater. The Plexxi Compatibility Matrix contains versionspecific software and hardware support information as well as cable and transceiver support information. Plexxi Release Notes for Switch-Control Plexxi Switch Administrator Guide using Linux and Plexxi CLI for applicable release Plexxi Control Online Help is available while logged into the Plexxi Control UI Except for the online help, this documentation is available on the Resources > Technical Publications page of Contacting Plexxi Support Plexxi Support services are available to answer questions and to make sure your software and hardware operate properly. Contact Plexxi Support at: support@plexxi.com (US/Canada toll-free) (US/International) 5 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

6 1 About Plexxi Control High Availability Plexxi Control and the Postgres database (DB) can be made highly available to protect from hardware or software failures. High Availability (HA) is accomplished by configuring and running multiple nodes that contain instances of Plexxi Control software, Postgres database (DB), and high availability clustering software. Plexxi requires three Plexxi Control/DB nodes as shown below for high availability of both Plexxi Control and the Postgres database. Plexxi Control runs as the master on one node and as a slave on the other nodes. Similarly, the Postgres DB runs as the master on one node and as a slave on the other nodes. HA clustering software monitors the health and availability of the nodes and resources in the cluster. If either the Plexxi Control master or the Postgres master fails for any reason, one of the slaves is promoted to master. On the node running the Plexxi Control master, there is also a Plexxi Control Virtual IP (VIP) instance running. The VIP runs ONLY on the node that the master is running on. Similarly, the node running the Postgres master also contains a running Postgres VIP instance. The management host is used to install, configure and manage the HA nodes. 6 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

7 HA Clustering Software Plexxi utilizes the CentOS cluster software stack to provide its high availability solution for Plexxi Control and Postgres. The CentOS cluster software stack's main components include Pacemaker which is a resource manager that manages the cluster resources and Corosync (Cluster Membership Layer) which manages the cluster membership layer, monitors availability of the nodes in the cluster, and handles split-brain conditions through use of quorum (policy) and STONITH (Shoot The Other Node In The Head). Management Node for HA For HA, you need to set up a management host by installing the Plexxi Control Installation and Configuration (PCIC) Python package. This package is built on top of Ansible, which PCIC uses to install, configure, upgrade, and manage Plexxi Control and the Postgres database on each node in the HA configuration. Ansible uses SSH to communicate with the remote nodes. PCIC in conjunction with Ansible automates software deployment, provisioning, configuration and management. As part of the setup, the HA clustering software (Pacemaker and Corosync) is installed and configured as well as Plexxi Control and Postgres. Pacemaker will start and manage Plexxi Control, Postgres and their respective Virtual IPs (VIPs). When Ansible is not managing nodes, it does not consume resources. Plexxi Control HA For HA, Ansible sets up and runs Plexxi Control on three separate nodes in an active/active configuration where Plexxi Control runs as the master on one node and as a warm standby slave on the other nodes. This enables a standby Plexxi Control instance to take over in seconds if the master node fails. Database Nodes for HA For database HA, multiple Postgres database instances must be installed and configured on the three nodes in the cluster. Virtual IP (VIP) Two VIPs are configured; one for Postgres and one for Plexxi Control. Each VIP runs on only a single node. The VIP for Postgres always runs on the node that is running the Postgres master. Similarly, the Control VIP always runs on the node that contains the Control master. Pacemaker is configured to ensure that the VIPs always follow their masters. Therefore, if a slave Plexxi Control or Postgres instance takes over as master, the VIP for the new Control or Postgres master will be started and run on the node now running the new master. Synchronization All HA nodes are synchronized such that any configuration or data changes made on the master node are replicated on the slave nodes. 7 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

8 2 Installation Considerations HA System and Software Prerequisites For HA, three nodes need to be created on either hardware servers or VMs (virtual machines). PlexxiControl Linux OS ISO, which will be installed on each node HA management node, MAC or Linux with Python 2 (versions 2.6 or 2.7) or Python 3 (versions 3.5 and higher) installed. Ansible Requirements Host Requirements To support PCIC, Ansible and Python, the management host must meet the following minimum requirements: 2 CPU cores 4 GB RAM 20 GB disk space Operating System Linux (CentOS recommended) Plexxi Control Requirements Plexxi Control runs as a Java application inside a Tomcat server in a Linux environment. Host Requirements To support Plexxi Control, the Plexxi Control virtual machine (VM) must meet the following minimum requirements: 4 CPU cores 8 GB RAM 200 GB disk space Supported VM Hypervisors for Plexxi Control VMware ESXi IMPORTANT: If you need to install Plexxi Control on a different hypervisor, contact Plexxi Support before attempting the installation. Operating System The installation deploys a Linux (CentOS) VM. 8 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

9 Security Considerations The Plexxi Control installation disables SELinux and SSH access to the root account and prompts you to enter a new root password on the first boot. You should also take additional measures as needed to secure the installation. See the Red Hat Enterprise Linux 6 Security Guide for more information: Secure Passwords A secure Linux Administrator password to be configured after installing Plexxi Control. A secure Plexxi Connect Administrator password to be configured after installing Plexxi Con trol. 9 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

10 3 Download Files from Plexxi 1. Log into and navigate to the HA subdirectory under the download directory for the version of Plexxi Control software. 2. Download the following files which will be used to install and configure HA: Plexxi Control Linux ISO installer file, plexxicontrol-linux-el6-offline-datestamp.iso Base Inventory file and stonith_params.yml file. Release Notes and other Plexxi documentation as needed. 10 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

11 4 Install PCIC on the Management Host PCIC (Plexxi Control Installation and Configuration) is a Python package that is installed on a Linux or macos machine running Python. It is recommended that it be installed into a Python virtual environment. Create a Python Virtual Environment If the Python virtual environment is not installed, virtualenv can be installed using the pip Python package manager as follows: > pip install virtualenv For additional instructions, refer to: Install PCIC in the Virtual Environment It is recommended that you set up a Python virtual environment to run PCIC within. This is accomplished by creating an environment with the virtualenv command and then sourcing the environment into your current bash shell. > virtualenv ~/pcic-venv > source ~/pcic-venv/bin/activate > pip install upgrade pip setuputils > pip install pcic To access the PCIC (Plexxi Control Installation and Configuration) script, pip needs to be configured to access the Plexxi PyPI server where the Python package is hosted. A configuration to allow access is shown below. pip.conf [global] Timeout = 60 index-url = Note: If you are using macos , you will also need to install the sshpass utility and the Python passlib package. There are multiple ways to install sshpass; the easiest is likely via brew ( The passlib package can then be installed with pip in your virtual environment. For example: > brew install > source ~/pcic-venv/bin/activate > pip install passlib Generate the SSH Key Ansible is used by PCIC to install and configure the nodes. Part of the initial prep that Ansible will perform on Plexxi Control HA nodes will be to create an Ansible user and enable key-based SSH access to it to perform subsequent configuration. It is recommended that an Ansible-specific key pair be generated for use. > ssh-keygen -f $HOME/.ssh/plexxi-control-ansible -t rsa -N '' 11 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

12 5 Create HA VMs and Install Linux CentOS on each VM Prepare the Plexxi Control Nodes To enable the Ansible-based installation of Plexxi Control, the base operating system must be installed on the nodes from the base Control CentOS/EL 6 Plexxi ISO. Set up the system as usual, configuring networking with either DHCP or static IP addresses. Take note of these addresses as they will be used in the inventory files used by PCIC/Ansible. This will be described in a subsequent section. Create a VM for Each Node 1. Create a Virtual Machine (VM) on which to install and run Plexxi Control. Adhere to the previouslyspecified requirements for memory, CPU, and disk. 2. Upload the Plexxi Control Linux ISO installer file to the disk_images directory of the ESXi host datastore. 3. Configure the VM to boot from the Plexxi Control Linux ISO installer file. Install Linux 1. Power ON the VM. 2. Immediately launch a remote VM console. The Linux installer automatically launches and the installer main menu opens in the Console. IMPORTANT: A timer counts down and the Static IP installation will begin automatically at the timeout. To select either the DHCP installation or the Troubleshooting menu, before the countdown completes, use the down-arrow and up-arrow keys to select the option, and then press Enter. The following is a sample main installer screen: Install Plexxi Control (Static IP): Static IP is the Default and recommended option. Use this option to install Plexxi Control using a statically assigned IP address. You will provide a Hostname, Domain name, IP Address, Netmask, Gateway, Name Servers (DNS), and optionally a VLAN ID and NTP server(s). Install Plexxi Control (DHCP): Use this installation option to install Plexxi Control and use DHCP 12 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

13 to assign the Hostname, Domain name, and optionally NTP server(s). Troubleshooting: Open the Troubleshooting menu. 3. Either press Enter to immediately launch the installation using Static IP or select DHCP installation and press Enter. Progress messages appear as the installation progresses. 4. Complete the following network information, then when finished, select OK and press Enter. For Static IP Complete the fields as follows: Hostname: Enter the host name for the VM. Domain name: Enter the domain name for the domain in which the host resides. Enter the domain name only, not a fully qualified domain name. If the network/deployment environment does not have a domain, you can specify something like "localdomain" as the domain name. NTP Servers: Optional: Enter NTP server names or IP addresses. IP Address: Enter a valid IP address for this Plexxi Control host. Netmask: Enter the network mask using either dot-decimal or CIDR (omit the slash) notation. Gateway: Enter a valid gateway address. VLAN ID: Enter a VLAN ID (1-4095) or 0 (default) if not using a VLAN. WARNING: The VLAN ID field should only be used when: o o software is being installed on 'bare metal' (non-hypervisor) servers. management traffic must be sent over a tagged VLAN. If your deployment does not meet these criteria, do not change the default value (0). Name Servers: Enter the IP addresses for DNS name servers, separated by spaces. 13 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

14 For DHCP Complete the fields as follows: Hostname: Enter the host name for the VM. Domain name: Enter the domain name for the domain in which the host resides. Enter the domain name only, not a fully qualified domain name. If the network/deployment environment does not have a domain, you can specify something like "localdomain" as the domain name. NTP Servers: Optional: Enter NTP server names or IP addresses. Error Notifications If you enter an invalid value in a parameter, when you click OK and Enter to continue, you will be notified with an error message in Red text. Examples include: ERROR: Gateway is not on the same network. ERROR: Nameserver sample.company.com is not a valid IP address. ERROR: Invalid hostname vm1_company.com. In this case, the hostname contains an invalid character (underscore). 5. You will be prompted to accept the Oracle End User License agreement. To accept the agreement, press Tab to select the Accept button, and then press Enter. 14 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

15 6. The Plexxi End User License agreement opens. To accept the agreement, press Tab to select the Accept button, and then press Enter. Progress messages appear as the installation progresses. The installation will take a few minutes. For example: When the installation completes successfully, the Installation completed Remove Install media screen appears. 7. Eject the installation media (disk or ISO image). IMPORTANT: If you do not eject the installation media and/or set the startup device to another drive, the host (VM) could boot back into the installer menu again. 8. Press Tab to select OK, and then press Enter. The system reboots. Continue You will be prompted to change the Linux root user password. Continue with the next section. 15 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

16 Changing the Linux Root User Password After installing Linux for Plexxi Control, when the Plexxi Control server/vm first boots, you are prompted to change the root user password. Until this step is completed, the Plexxi Control server will not be fully functional. Enter a new Linux root user password, then as prompted to confirm, re-enter the password. Continue The Select Action screen opens. Continue with the next section. Selecting an Action After entering a new Linux root user password, the following Select Action screen opens. Continue In this screen, select the next action to continue: IMPORTANT: Unless you need to change the network settings that you previously entered, this screen can be skipped by selecting Quit. Device configuration Optional. Used to configure interface devices such as eth0. Continue with the section, Configuring Network Settings for a Device on page 28. DNS configuration Optional. To configure DNS, continue with the section, Configuring DNS on page 29. Save&Quit or Quit Save&Quit saves configuration information and completes the installation. Quit completes the installation without saving configuration changes made since the last reboot. For both Save&Quit or Quit, continue with the next section to select the time zone. 16 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

17 Selecting the Time Zone Selecting Save&Quit at the Select Action screen opens the Configure Timezones screen: In this screen, use the up-arrow and down-arrow to select the time zone, click Tab to select OK, and then press Enter. Plexxi recommends using the UTC time zone. Continue The Authentication Configuration screen opens. Continue with the next section to complete the installation. Completing the Installation Selecting OK in the Configure Timezones screen opens the following Authentication Configuration screen. 1. Press Tab to select Next and then press Enter to continue. Do NOT make any configuration changes on this screen. The installation progress displays: The host completes booting and the localhost login prompt appears. The installation is complete. 17 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

18 6 Set Up the Ansible Files Several configuration files on the management host need to be set up and several optional command line parameters need to be specified to control the installation. Inventory file Ansible uses an inventory file to specify the hosts and groups of hosts that will be accessed by the configuration steps in the playbooks. Inventory files can be located in a read/write directory. Its location must be specified as a command line parameter when running the various PCIC commands. The file can be in a number of different formats. For Plexxi Control, the INI-like format works well. Only one section, defining an Ansible group of hosts, is needed. It should be named "plexxi -control". In this section (group) a group of hosts should then be specified. For a non-ha configuration, only one host needs to be specified, as shown below. Inventory File - Single Instance [plexxi-control] pc-node = ansible_host= Inventory File - High Availability For a High Availability configuration, three or more hosts must be specified. The following is an example of what that section will look like in the inventory file. [plexxi-control] pc-node1 = ansible_host= pc-node2 = ansible_host= pc-node3 = ansible_host= Inventory File: HA and Ansible Playbook Variables Ansible provides multiple ways to specify variables. However, to install and configure Plexxi Control, specify the variables in the Inventory file. The following variables need to be specified to direct the configuration steps in the Ansible playbooks: ansible_ssh_private_key_file pathname to the private key file that will be installed in the "ansible" user that is created on the host(s). ansible_ssh_public_key_file pathname to the public key file that will be used to ssh into the "ansible" user on the host(s). For an HA configuration, the following variables are required: pgsql_vip the Virtual IP, created on the master host, to access PostgreSQL. plexxi_control_vip the Virtual IP, created on the master host, to access Plexxi Control. plexxi_control_vip_hostname hostname for the Control VIP (as specified in DNS) needed to do HTTPs to Plexxi Control. Needs to be the Fully Qualified Domain Name (FQDN). 18 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

19 Example: INI Inventory File Note: You must provide your custom values for all bold-italic variables in the following file. IMPORTANT: The pacemaker_cluster_name must be 15 characters or less or else it will fail to start. [plexxi-control] node1 = ansible_host= node2 = ansible_host= node3 = ansible_host= [plexxi-control:vars] pgsql_vip = vip_cidr_netmask=16 plexxi_control_vip_hostname=control-host.mycompany.com pacemaker_cluster_name=ha_cluster_1 plexxi_control_vip = ansible_ssh_public_key_file = '~/.ssh/ansible.pub ansible_ssh_private_key_file = '~/.ssh/ansible' Configure STONITH To configure STONITH, fencing needs to be specified in a YAML file. Fencing is a very important concept in computer clusters for high availability. With fencing, even when the cluster does not know what is happening on a node, we can make sure that that node does not run important resources. Pacemaker fencing uses Shoot the Other Node in the Head (STONITH). To configure STONITH for Plexxi Control HA through Ansible, fencing plugin(s) need to be configured. The plugins are provided with the Pacemaker software and control a range of fencing devices such as power controllers, lights-out management software, and virtualization systems such as ESXi/vCenter. For Plexxi Control, fencing plugins are configured within a YAML file as shown below for a single ESXi/vCenter host. STONITH Parameters, Single Host Note: You must provide your custom Stonith values for all bold-italic variables in the following file.... pacemaker_stonith_resources:... - name: "host-fence" class: 'stonith' type: "fence_vmware_soap" options: ipaddr: ' host.customer.com' login: 'root' passwd: 'password' ipport: 44 ssl_insecure: 1 inet4_only: 1 action: 'off' pcmk_host_map: 'pc-ha4=pc-ha4,pc-ha5=pc-ha5,pc-ha6=pc-ha6' pcmk_host_check: 'static-list' pcmk_host_list: 'pc-ha4,pc-ha5,pc-ha6' power_wait: 3 op: - action: monitor options: interval: 60s 19 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

20 STONITH Parameters - multiple hosts If there is a need for more than one fencing plugin to be configured, either for a different class of fencing devices or for multiple devices, multiple unique elements can be configured as show below.... pacemaker_stonith_resources:... - name: "host1-fence" class: 'stonith' type: "fence_vmware_soap" options: ipaddr: 'host1.customer.com' login: 'root' passwd: 'password' ipport: 443 ssl_insecure: 1 inet4_only: 1 action: 'off' pcmk_host_map: 'pc-ha4=pc-ha4,pc-ha5=pc-ha5' pcmk_host_check: 'static-list' pcmk_host_list: 'pc-ha4,pc-ha5' power_wait: 3 op: - action: monitor options: interval: 60s - name: " host2-fence" class: 'stonith' type: "fence_vmware_soap" options: ipaddr: ' host2.customer.com' login: 'root' passwd: 'password' ipport: 443 ssl_insecure: 1 inet4_only: 1 action: 'off' pcmk_host_map: 'pc-ha6=pc-ha6' pcmk_host_check: 'static-list' pcmk_host_list: 'pc-ha6' power_wait: 3 op: - action: monitor options: interval: 60s To enable STONITH, in the Inventory file, set the stonith_param_file parameter, where the value is the path in which the STONITH file can be accessed. Inventory File with STONITH [plexxi-control] node1 = ansible_host= node2 = ansible_host= node3 = ansible_host= obm=ipmi [plexxi-control:vars] pgsql_vip = vip_cidr_netmask=16 pacemaker_cluster_name=cluster1 plexxi_control_vip = plexxi_control_vip_hostname = my_control_hostname.plexxi.com ansible_ssh_public_key_file = '~/.ssh/ansible.pub' ansible_ssh_private_key_file = '~/.ssh/ansible' stonith_param_file=~/stonith_params.yml 20 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

21 7 Plexxi Control High Availability Operations Install Plexxi Control High Availability The Plexxi Control software, including its dependencies, is installed by running the PCIC (Plexxi Control Installation and Configuration) script. The PCIC script calls an Ansible playbook that will configure the nodes and install and configure the software. To install Plexxi Control in an HA environment, do the following: 1. On the management host, run the pcic install command, specifying both the name of an inventory file and a Plexxi Control version number: > pcic install inventory-file version For example: > pcic install ~/ha_inventory_file The Plexxi Control HA installation begins and completes. Next, the Tomcat application server starts up. Allow 2-3 minutes for the Plexxi Control application to start. Plexxi Control is now installed. 3. When finished, verify the installation. In a Web browser, log into the Plexxi Control UI as administrator using your Plexxi Control administrator account login and password. The URL format is: IMPORTANT: When connecting to Plexxi Control on port 8443, you must use the server's FQDN (Fully Qualified Domain Name). 4. Verify the Plexxi Control UI version and that the UI opens successfully. Upgrade Plexxi Control High Availability If Plexxi Control needs to be upgraded in an HA environment, you run the pcic upgrade command. 1. On the management host, run the pcic upgrade command, specifying both the name of an inventory file and a Plexxi Control version number: > pcic upgrade inventory-file version For example: > pcic upgrade ~/ha_inventory_file The Plexxi Control HA upgrade begins. When the upgrade successfully completes, you will see the following message: Plexxi Control software was successfully upgraded to version #.#.#! Next, the Tomcat application server starts up. Allow 2-3 minutes for the Plexxi Control application to start. Plexxi Control is now upgraded. 3. When finished, verify the upgrade. In a Web browser, log into the Plexxi Control UI as administrator using your Plexxi Control administrator account login and password. The URL format is: 21 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

22 IMPORTANT: When connecting to Plexxi Control on port 8443, you must use the server's FQDN (Fully Qualified Domain Name). 4. Verify the Plexxi Control UI version and that the UI opens successfully. Update Plexxi Control High Availability If Plexxi Control needs to be updated in an HA environment, you run the pcic update command. The pcic update command does not take a version or release number. It automatically updates Plexxi Control to the latest version of the currently installed release. On the management host, run the pcic update command, specifying the name of an inventory file: > pcic update inventory-file For example: > pcic update ~/ha_inventory_file The Plexxi Control HA update begins and completes. Uninstall Plexxi Control High Availability If Plexxi Control High Availability needs to be removed from an environment, you run the pcic uninstall command. On the management host, run the pcic uninstall command, specifying the name of an inventory file: > pcic uninstall inventory-file For example: > pcic uninstall ~/ha_inventory_file The Plexxi Control HA uninstall begins and completes. 22 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

23 8 Verifying High Availability Pacemaker To check the active/standby status of nodes within the Pacemaker cluster, run the pcs status command (as root or via sudo) from any of the hosts in the cluster (as show below). ~]# pcs status Cluster name: pc2 Stack: cman Current DC: pc-ha5 (version el6-e174ec8) - partition with quorum Last updated: Thu Jan 18 09:25: Last change: Thu Jan 18 09:24: by root via crm_attribute o 3 nodes and 11 resources configured Online: [ pc-ha4 pc-ha5 pc-ha6 ] Full list of resources: pgsql-vip (ocf::heartbeat:ipaddr2): Started pc-ha5 plexxidb-backup (ocf::heartbeat:symlink): Started pc-ha5 plexxidb-maint (ocf::heartbeat:symlink): Started pc-ha5 plexxidb-server (ocf::heartbeat:symlink): Started pc-ha5 Master/Slave Set: pgsql-master [pgsql] Masters: [ pc-ha5 ] Slaves: [ pc-ha4 pc-ha6 ] plexxi-control-vip (ocf::heartbeat:ipaddr2): Started pc-ha6 Master/Slave Set: plexxi-control-master [plexxi-control] Masters: [ pc-ha6 ] Slaves: [ pc-ha4 pc-ha5 ] Failed Actions: * pgsql_monitor_16000 on pc-ha6 'unknown error' (1): call=382, status=timed Out, exitreason='none', last-rc-change='thu Jan 18 09:17: ', queued=0ms, exec=10002ms * pgsql_monitor_16000 on pc-ha5 'unknown error' (1): call=379, status=timed Out, exitreason='none', last-rc-change='thu Jan 18 09:17: ', queued=0ms, exec=10003ms Daemon Status: cman: active/disabled corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled 23 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

24 Verifying PostgreSQL HA To check the replication status of PostgreSQL database HA, run the following psql command (as the postgres user) on the node identified as the master within the Pacemaker Master/Slave set labeled 'pgsqlmaster' (as shown below). postgresql replication status > sudo -u postgres psql -c 'select application_name as host, client_addr, client_port, backend_start, backend_xmin, state, sync_priority, sync_state from pg_stat_replication;' host client_addr client_port backend_start backend_xmin state sync_priority sync_state pc-ha :18: streaming 3 potential pc-ha :24: streaming 1 sync The primary server provides sync_priority and sync_state to all managed standby servers, and treats each standby server depending on its respective values: sync_priority indicates the priority of the standby server in synchronous-mode and is a fixed value. The smaller value shows the higher priority, while 0 is the special value that means "in asynchronous-mode". Priorities of standby servers are given in the order listed in the primary's configuration parameter synchronous_standby_names. sync_state is the state of the standby server. It is variable according to the running status of all standby servers and the individual priority. The possible states are: o o o Sync is the state of synchronous-standby server of the highest priority among all working standbys (except asynchronous-servers). Potential is the state of spare synchronous-standby server of the second or lower priority among all working standbys (except asynchronous-servers). If the synchronous-standby has failed, it will be replaced with the highest priority standby within the potential ones. Async is the state of asynchronous-standby server, and this state is fixed. The primary server treats asynchronous-standbys in the same way as potential standbys except that their sync_state can never be 'sync' or 'potential'. 24 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

25 Verifying Plexxi Control High Availability After installation, an instance of Plexxi Control will be running on every node; one as the master and the others as slaves. The following file informs each Plexxi Control instance what role it should assume: /var/run/tomcat/plexxi-control.state Do not edit or delete this file! You can view it to determine which role Pacemaker wishes Plexxi Control to take: master or slave. In addition, the PlexxiCore log file (/var/log/tomcat/plexxi/plexxicore.log) contains the following information: 13:31: [localhost-startstop-1] INFO c.p.f.engine.enginemanagerimpl - Engine Manager: start Enabled: true Active : false HA mode: true For HA mode, the HA mode variable must be true. 25 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

26 9 Administering Plexxi Control Hosts Changing Default Linux Passwords on a New VM Plexxi recommends that, soon after deploying a new Plexxi Control VM, you connect to the new VM and change the default Linux passwords for the root and plexxi user accounts. IMPORTANT: Please ensure that you record these passwords in case they are required for futu re Controller server access. Plexxi Support will be unable to assist with access to your Controller if you change the default passwords and do not know the new passwords. To change the default Linux passwords for the root and plexxi user accounts: 1. Log into Linux (SSH, console) as user plexxi using the default password: plexxi 2. To change the plexxi user password, enter the passwd command and enter the new password twice as prompted: $ passwd 3. Access the root user account. When prompted for a password by the "sudo" command, enter the plexxi" account password that you just set above. $ sudo su 4. To change the root user password, enter the following command and enter the new passwords as prompted: # passwd 5. Exit the root user: # exit $ Changing the Plexxi Control Password Note: To change the administrator password or the password of any user other than yourself, you must have Administrator privileges. Any user can change their own password once logged into Plexxi Control. To change the password for any Plexxi Control user: 1. Open the Plexxi Control UI. In a Web browser, enter the Plexxi Control URL. The URL format is: IMPORTANT: When connecting to Plexxi Control on port 8443, you must use the server's FQDN (Fully Qualified Domain Name). 2. Log into Plexxi Control as administrator. If you are logging into Plexxi Control for the first time, the default credentials are: Username: admin Password: plexxi 26 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

27 3. In the Plexxi Control header, select the Operations icon > Configure > Users. The Manage Users window opens. For example: 4. Select the user and click Edit. The Edit User window opens. 5. In the Password and Confirm Password fields, enter and re-enter the new password. For example: 6. Click OK. The new password will be in effect. 7. Click Close to exit the Manage Users window. The message contains the URL of the new Plexxi Control UI. For example: 2. When finished, verify the Plexxi Control installation. In a Web browser, enter the Plexxi Control URL that was displayed by the Plexxi Control installer. The URL format is: 27 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

28 IMPORTANT: When connecting to Plexxi Control on port 8443, you must use the server's FQDN (Fully Qualified Domain Name). Log into Plexxi Control as administrator using the default credentials: Username: admin Password: plexxi Verify that the Plexxi Control UI opens successfully. 3. Also, you can log into Linux on the Plexxi Control VM as either: User: root and the new Linux root password set earlier in this procedure. User plexxi with the default password plexxi. 4. You can test remote access to Linux as described in the next section. 5. Plexxi recommends that while logged into the Plexxi Control UI as administrator, you change the Plexxi Control password to a secure password. Refer to the section Changing the Plexxi Control Password on page Plexxi recommends that you connect to the new VM and change the default Linux passwords for the root and plexxi user accounts. Refer to the section Changing Default Linux Passwords on a New VM on page The Plexxi Control installation is complete. Disregard all other installation chapters. To configure authentication to the Plexxi switches, refer to the Plexxi Switch Administrator Guide using Linux and Plexxi CLI. To configure advanced authentication for the Control VM system (i.e., UNIX logins to the machine using LDAP, NIS, or directory service for account/password management, etc.) refer to the RedHat Enterprise Linux documentation. Testing Remote Access to the Linux Host Plexxi recommends that you test remote (SSH) access to the Linux host by doing the following: 1. Run the ifconfig command to determine the IP address of the Linux host. The device name is hardware-dependent. If you are using a VM, it will be eth0, however, on a bare-metal server it may be different (though is usually either eth0 or eth1). 2. From another system, attempt an SSH connection to your server's IP address. Note: root logins via SSH are disabled by default. You will need to login using the plexxi user account, using the password plexxi. Optional Configurations The following device and DNS configuration sections are optional. These settings were previously entered during the installation. Configuring Network Settings for a Device IMPORTANT: The settings in this section were previously entered during the installation. You can use these optional screens if you want to change network settings that you previously entered. Otherwise, skip this section. In the Select Action screen, if you select Device configuration, the Select a Device screen opens. From this screen, you can select network devices on the Plexxi Control server/vm to configure. In this example eth0 is being configured: 28 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

29 1. Press tab to select a device and then press Enter. The Network Configuration screen opens to configure the selected device. 2. Edit the device configuration as needed. 3. Press Tab to select OK and then press Enter (or select Cancel to cancel the changes). You are returned to the Select a Device screen. 4. In the Select a Device screen: To save changes and return to the Select Action screen, press Tab to select Save, and then press Enter. To cancel and return to the Select Action screen. press Tab to select Cancel, and then press Enter. 5. Refer to Selecting an Action on page 16. Configuring DNS IMPORTANT: The settings in this section were previously entered during the installation. You can use this optional screen if you want to change DNS network settings that you previously entered. Otherwise, skip this section. 1. In the Select Action screen, press the Down arrow to select DNS configuration and press Enter. The DNS configuration screen appears. 29 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0

30 2. Make the edits you want and press Tab to select OK. (Or Cancel to cancel the operation.) 3. Press Spacebar. You are returned to the Select Action screen. 4. Select Quit to exit or Save&Quit to exit. The Authentication Configuration screen appears. See Authentication Configuration for details. Troubleshooting Menu Selecting Troubleshooting from the main menu opens the following Troubleshooting menu: Rescue an existing system: Rescue a previously installed Linux system. Run a memory test: Use this utility to test the system memory and verify that it is working correctly. Boot from local drive: Attempt to boot the system from the local drive, using the boot order as defined in the system BIOS. Return to main menu: Return to the main installer menu. 30 Plexxi Control High Availability: Installation, Upgrade and Administration Guide, 4.0.0