Control Center Installation Guide for High-Availability Deployments

Transcription

1 Control Center Instlltion Guide for High-Avilility Deployments Relese Zenoss, Inc.

2 Control Center Instlltion Guide for High-Avilility Deployments Copyright 2017 Zenoss, Inc. All rights reserved. Zenoss nd the Zenoss logo re trdemrks or registered trdemrks of Zenoss, Inc., in the United Sttes nd other countries. All other trdemrks, logos, nd service mrks re the property of Zenoss or other third prties. Use of these mrks is prohiited without the express written consent of Zenoss, Inc., or the third-prty owner. Linux is registered trdemrk of Linus Torvlds. All other compnies nd products mentioned re trdemrks nd property of their respective owners. Prt Numer: Zenoss, Inc Four Points Drive Bldg 1 - Suite 300 Austin, Texs

3 Contents Aout this guide...6 Chpter 1: Plnning high-vilility deployment...8 Aout high-vilility deployments...8 HA cluster configurtion options... 9 Fencing recommendtions... 9 Mster host storge requirements...10 Mster host resource requirements...10 Key vriles used in this guide Chpter 2: Instlling mster host Verifying cndidte host resources Disling consistent network device nming...14 Prepring the mster host operting system...15 Instlling required softwre nd imges Configuring privte mster NTP server Instlling Docker Engine Instlling Control Center Configuring Docker Engine nd loding imges...23 Chpter 3: Configuring Control Center on mster host nodes Control Center mintennce scripts on the mster host...26 User ccess control Configuring the se size device for tennt dt storge Setting the host role to mster Configuring offline use...30 Configuring endpoints...30 Mster host configurtion vriles Universl configurtion vriles...34 Chpter 4: Configuring DRBD...36 Instlling cluster mngement Configuring Logicl Volume Mnger Configuring DRBD...38 Initilizing DRBD...40 Chpter 5: Cluster mngement softwre...42 Creting the cluster in stndy mode...42 Property nd resource options Defining resources Chpter 6: Verifiction procedures...48 Verifying the DRBD configurtion...48 Verifying the Pcemker configurtion

4 Verifying the Control Center configurtion...49 Verifying cluster strtup Verifying cluster filover...51 Chpter 7: Configuring uthentiction on mster host nodes...53 Creting new resource pools Adding mster nodes to their resource pool Chpter 8: Instlling delegte hosts...55 Verifying cndidte host resources Delegte host storge requirements...57 Instlling required softwre nd imges Configuring NTP clients...59 Disling consistent network device nming...60 Prepring delegte host...61 Instlling Docker Engine Instlling Control Center Configuring Docker Chpter 9: Configuring nd strting delegte hosts...68 Control Center mintennce scripts on delegte hosts...68 Enling use of the commnd-line interfce...68 Setting the host role to delegte...69 Configuring offline use...69 Configuring endpoints...70 Delegte host configurtion vriles Universl configurtion vriles...72 Strting Control Center...74 Delegte host uthentiction...74 Chpter 10: Configuring ZooKeeper ensemle ZooKeeper nd Control Center Understnding the configurtion process Configuring mster host nodes s ZooKeeper node Configuring delegte host A s ZooKeeper node...80 Configuring delegte host B s ZooKeeper node Instlling the ZooKeeper imge for Docker Engine Strting ZooKeeper ensemle Updting delegte hosts...85 Appendix A: Strting nd stopping Control Center deployments Stopping Control Center...87 Strting Control Center...90 Appendix B: Storge mngement utility...92 serviced-storge...92 Appendix C: Control Center configurtion vriles

5 Best prctices for configurtion files Control Center configurtion file

6 Control Center Instlltion Guide for High-Avilility Deployments Aout this guide Control Center Instlltion Guide for High-Avilility Deployments provides detiled procedures for instlling nd configuring Control Center cluster in high-vilility deployment. Relted pulictions Title Control Center Relese Notes Control Center Plnning Guide Control Center Instlltion Guide Control Center Instlltion Guide for High-Avilility Deployments Control Center Reference Guide Control Center Upgrde Guide Control Center Upgrde Guide for High- Avilility Deployments Description Descries known issues, fixed issues, nd lte-reking informtion not included in other pulictions. Provides oth generl nd specific informtion out prepring to deploy Control Center cluster. Provides detiled procedures for instlling nd configuring Control Center cluster. Provides detiled procedures for instlling nd configuring Control Center in high-vilility deployment. Provides informtion nd procedures for mnging Control Center. This informtion is lso ville s online help in the Control Center rowser interfce. Provides detiled procedures for updting Control Center deployment to the ltest relese. Provides detiled procedures for updting high-vilility deployment of Control Center to the ltest relese. Documenttion feedck To provide feedck out this document, or to report n error or omission, plese send n emil to docs@controlcenter.io. In the emil, plese include the document title nd prt numer, nd s much informtion s possile out the context of your feedck. The prt numer ppers t the end of the list of trdemrks, t the front of this guide. Supported clients nd rowsers The following tle identifies the supported comintions of client operting systems nd we rowsers. Client OS Windows 7 nd 8.1 Windows 10 Windows Server 2012 R2 Supported Browsers Internet Explorer 11 (Enterprise mode only; comptiility mode is not supported.) Internet Explorer 11 (Enterprise mode only; comptiility mode is not supported.) Firefox 50 nd lter Chrome 54 nd lter Microsoft Edge Firefox 30 Chrome 36 6

7 Aout this guide Client OS Mcintosh OS/X 10.9 Supported Browsers Firefox 30 nd ove Chrome 36 nd ove Uuntu LTS Firefox 30 nd ove Chrome 37 nd ove Red Ht Enterprise Linux 6.5, CentOS 6.5 Firefox 30 nd ove Chrome 37 nd ove Chnge history The following list ssocites document prt numers nd the importnt chnges to this guide since the previous relese. Some of the chnges involve fetures or content, ut others do not. For informtion out new or chnged fetures, refer to the Control Center Relese Notes Move the step for loding Docker Engine imges to the next procedure Initil relese (1.3.1). 7

8 Control Center Instlltion Guide for High-Avilility Deployments Plnning high-vilility deployment 1 This chpter provides informtion out plnning high-vilility deployment of Control Center, nd out prepring to crete high-vilility deployment. Before proceeding, plese red the Control Center Plnning Guide. For optiml results, review the contents of this guide thoroughly efore performing n instlltion. Note The procedures in this guide descrie how to configure deployment tht does not hve internet ccess. You my crete deployment tht does hve internet ccess; the procedures ssume tht the deployment does not hve ccess. Aout high-vilility deployments Control Center cn e deployed in high-vilility (HA) configurtion to minimize downtime cused y the filure of hrdwre components or operting system services in Control Center mster host. An HA deployment cn e configured in vriety of wys, including the following: Active-Active, non geo-diverse Active-Pssive, geo-diverse Active-Pssive, non geo-diverse (works with Control Center) The Active-Pssive configurtion tht works with Control Center: uses Pcemker, Corosync, nd Distriuted Replicted Block Device (DRBD) to mnge the HA cluster includes two or more identicl mster hosts in the HA cluster one primry node nd one or more secondry nodes, which tke over s the Control Center mster host if the primry node ecomes unville requires minimum of two mster hosts nd two delegte hosts provides no protection ginst fcility ctstrophe ecuse the HA cluster nodes re locted in the sme fcility Note Zenoss supports DRBD 8.4, not DRBD 9.0. The procedures in this guide instll the ltest version of relese

9 Plnning high-vilility deployment HA cluster configurtion options The recommended deployment rchitecture requires two identicl dul-nic mchines for optiml disk synchroniztion nd network responsiveness. However, you cn deploy n HA cluster with single-nic servers if two identicl dul-nic mchines re not ville. Requirements for two identicl dul-nic servers Mster hosts: In seprte resource pool, you need two identicl hosts in the role of Control Center mster host; one host serves s the primry node nd the other s the secondry node. Provide two NICs on the HA cluster primry node nd two on the secondry node. On ech node, dedicte one NIC to network trffic tht is required for disk synchroniztion vi DRBD. Control Center nd ppliction trffic. Route trffic for ech NIC through seprte sunets. Delegte hosts: You need N+1 identicl hosts to serve s delegte hosts, where N is the numer of hosts needed to stisfy the performnce nd sclility requirements of the tennt. They do not need to e memers of the HA cluster ecuse, if host ecomes unville, Control Center restrts their services on other hosts. Deploy ppliction services on dedicted Control Center delegte hosts outside the HA cluster. Requirements for two single-nic servers Mster host: Configure two hosts for the Control Center mster host in n ctive/pssive cluster. Use the two hosts only to run Control Center. Do not use them to run ppliction services. For primry nd secondry nodes tht contin only one network-interfce crd (NIC), the network they use to communicte must support multicst. Delegte hosts: You need N+1 identicl hosts to serve s delegte hosts, where N is the numer of hosts needed to stisfy the performnce nd sclility requirements of the pool. (Only the Control Center mster hosts must e configured in n ctive/pssive cluster.) Fencing recommendtions Fencing is n utomted mens of isolting node tht ppers to e mlfunctioning, used to protect the integrity of the DRBD volumes. In test deployment of Control Center HA cluster, fencing is not necessry. However, on production HA clusters, fencing is criticl considertion. Work with your IT deprtment to implement the est fencing solution for your infrstructure. Employ technique tht ensures tht filed node in the cluster is completely stopped to void ppliction conflicts or conflicts with the cluster mngement softwre. When fencing is employed in the HA cluster, use two NICs per node. Before you configure nd enle fencing in your production environment: Ensure tht ll components re deployed. Verify opertion of the ppliction tht Control Center is mnging. In controlled scenrio, confirm sic cluster filover. 9

10 Control Center Instlltion Guide for High-Avilility Deployments If fencing method is not defined, when the cluster ttempts to fil over to the ckup node, the following error results: no method defined Plce the fencing device on the pulic network. (Pssing hertet communiction through privte network interfce is not recommended. Doing so requires complex fencing system tht is prone to issues. For more informtion, see Quorum Disk documenttion on the Red Ht wesite.) Using pulic network interfce enles helthy node to fence the unhelthy node, nd prevents the unhelthy node from fencing the helthy node. If hertet communictions pss through the pulic network nd the link for node goes down, the node with the down pulic network link cnnot communicte with the fencing device. Mster host storge requirements The following tle identifies the minimum locl storge requirements of Control Center mster host nodes in high-vilility deployments. The storge requirements re in ddition to the storge required for the host's operting system. Purpose Minimum size Description Docker dt 50GB (required) Locl, high-performnce storge. Control Center internl services dt 50GB (required) Locl, high-performnce storge formtted s n XFS filesystem. Appliction dt 200GB (suggested) Locl, high-performnce storge. Appliction metdt 1GB (required) Locl, high-performnce storge formtted s n XFS filesystem. Appliction dt ckups 150GB (suggested) Remote, XFS-comptile storge. This guide includes procedures for prepring the required lock storge res. Except for the Docker dt storge re, the underlying storge must e seprte rel or virtul devices tht DRBD cn replicte. The suggested minimum sizes for ppliction dt nd ppliction dt ckups should e replced with sizes tht meet your ppliction requirements. To clculte the pproprite sizes for these storge res, use the following guidelines: Appliction dt storge includes spce for oth dt nd snpshots. The defult se size for dt is 100GB, nd the recommended spce for snpshots is 100% of the se size. Adding the two yields the suggested minimum size of 200GB. For ppliction dt ckups, the recommended spce is 150% of the se size for dt. The suggested minimum size for ppliction dt ckups is 150GB. For improved reliility nd performnce, Zenoss strongly recommends using shred remote storge or network-ttched storge for ppliction dt ckups. This guide does not include instructions for mounting remote storge, ut does include step for creting the mount point. Mster host resource requirements The defult recommendtion for multi-host deployments is to use mster host for Control Center services only. In high-vilility deployments, some ppliction services perform more relily when they run on mster host nodes. In these cses, mster host nodes require dditionl RAM nd CPU resources. 10

11 Plnning high-vilility deployment Specificlly, Zenoss pplictions include dtse service tht performs est on mster host nodes. For more informtion, plese contct your Zenoss representtive. Key vriles used in this guide The following tles ssocite key fetures of high-vilility deployment with vriles used in this guide. Feture Pulic IP ddress of mster node (sttic; known to ll mchines in the Control Center cluster) Pulic hostnme of mster node (returned y unme; resolves to the pulic IP ddress) Privte IP ddress of mster node (sttic; dul-nic systems only) Privte hostnme of mster node (resolves to the privte IP ddress; dul-nic systems only) Vrile Nme Primry Node Primry-Pulic-IP Primry-Pulic-Nme Primry-Privte-IP Primry-Privte-Nme Secondry Node Secondry-Pulic-IP Secondry-Pulic-Nme Secondry-Privte-IP Secondry-Privte-Nme Feture Virtul IP ddress of the high-vilility cluster (sttic; known enterprise-wide) Virtul hostnme of the high-vilility cluster (known enterprise-wide) Mirrored storge for Control Center internl services dt Mirrored storge for ppliction metdt Mirrored storge for ppliction dt Vrile Nme HA-Virtul-IP HA-Virtul-Nme Isvcs-Storge Metdt-Storge App-Dt-Storge 11

12 Control Center Instlltion Guide for High-Avilility Deployments Instlling mster host 2 This chpter descries how to instll Control Center on Red Ht Enterprise Linux (RHEL) or CentOS host. The cndidte host must hve the CPU, RAM, nd storge resources required to serve s mster host node in Control Center cluster. Perform the procedures in this chpter on oth of the cndidte mster host nodes of high-vilility cluster. Verifying cndidte host resources Use this procedure to determine whether the hrdwre resources nd instlled operting system of host re sufficient to serve s Control Center mster host. 1 Log in to the cndidte host s root, or s user with superuser privileges. 2 Verify tht the host implements the 64-it version of the x86 instruction set. unme -m If the output is x86_64, the rchitecture is 64-it. Proceed to the next step If the output is i386/i486/i586/i686, the rchitecture is 32-it. Stop this procedure nd select different host. 3 Determine whether the instlled operting system relese is supported. ct /etc/redht-relese If the result includes 7.1, 7.2, or 7.3 proceed to the next step. If the result does not include 7.1, 7.2, or 7.3, select different host, nd then strt this procedure gin. 4 Determine whether the CPU resources re sufficient. Disply the totl numer of CPU cores. ct /proc/cpuinfo grep -Ec '^core id' Compre the ville resources with the requirements for Control Center mster host. For more informtion, refer to the Control Center Plnning Guide. 5 Determine whether the CPU resources support the AES instruction set. ct /proc/cpuinfo grep -Ec '^flgs.*es' 12

13 Instlling mster host For optiml performnce, the result of the preceding commnds must mtch the totl numer of CPU resources ville on the host. If the result is 0, performnce is severely degrded. If the result is 0 nd the cndidte host is virtul mchine, the mnging hypervisor my e configured in Hyper-V comptiility mode. Check the setting nd disle it, if possile, or select different host. 6 Determine whether the ville memory nd swp is sufficient. Disply the ville memory. free -h Compre the ville memory with the mount required for mster host in your deployment. For more informtion, refer to the Control Center Plnning Guide. 7 Ensure the host hs persistent numeric ID. Skip this step if you re instlling single-host deployment. Ech host in Control Center cluster must hve unique host ID, nd the ID must e persistent (not chnge when the host reoots). test -f /etc/hostid genhostid ; hostid Record the ID for comprison with other hosts in the cluster. 8 Verify tht nme resolution works on this host. hostnme -i If the result is not vlid IPv4 ddress, dd n entry for the host to the network nmeserver, or to /etc/ hosts. 9 Add n entry to /etc/hosts for loclhost, if necessry. Determine whether is mpped to loclhost. grep /etc/hosts grep loclhost If the preceding commnds return no result, perform the following sustep. Add n entry to /etc/hosts for loclhost. echo " loclhost" >> /etc/hosts 10 Updte the Linux kernel, if necessry. Determine which kernel version is instlled. unme -r If the result is lower thn el7.x86_64, perform the following sustep. Updte the kernel, nd then restrt the host. The following commnds require internet ccess or locl mirror of operting system pckges. yum mkecche fst && yum updte -y kernel && reoot 11 Disply the ville lock storge on the cndidte host. lslk -p --output=name,size,type,fstype,mountpoint Compre the results with the storge requirements descried in Mster host storge requirements on pge

14 Control Center Instlltion Guide for High-Avilility Deployments Disling consistent network device nming The consistent network device nming feture cn crete device nmes tht re too long for use with virtul IPs. This step disles the feture so tht ll network device nmes re eth followed y n integer. 1 Log in s root, or s user with superuser privileges. 2 Edit the GRUB 2 generl settings file. c Crete ckup copy of the settings file. cp /etc/defult/gru /etc/defult/gru.k Open /etc/defult/gru with text editor. Add kernel oot rguments to the vlue of the GRUB_CMDLINE_LINUX vrile. The rguments to dd re iosdevnme=0 nd net.ifnmes=0. Mke sure the rguments re etween the delimiter chrcters ("). d Sve the file, nd then close the text editor. 3 Determine whether the host is configured for UEFI or legcy oot mode. find /oot -mxdepth 1 -type d If the output includes /oot/efi, the host is configured for UEFI oot mode. If the output does not include /oot/efi, the host is configured for legcy oot mode. 4 Recrete the GRUB 2 oot configurtion file. For UEFI oot mode: gru2-mkconfig -o /oot/efi/efi/centos/gru.cfg For legcy oot mode: gru2-mkconfig -o /oot/gru2/gru.cfg 5 Crete n override for the defult rules policy of the dynmic device mngement demon. ln -s /dev/null /etc/udev/rules.d/80-net-nme-slot.rules 6 Chnge network device configurtions. Chnge directory to /etc/sysconfig/network-scripts. cd /etc/sysconfig/network-scripts List the network device configurtion files. c ls ifcfg-* grep -v ifcfg-lo The ifcfg-lo file is excluded ecuse it does not need to e chnged. Renme nd edit device files. For exmple, if you hve device file nmed ifcfg-enp0s3, use the mv commnd renme it to ifcfg-eth0, nd then chnge the vlues of the NAME nd DEVICE vriles inside the file to eth0. 7 Updte the defult route. Disply the current defult route. ip route show hed -n 1 14

15 Instlling mster host Exmple result, showing device enp0s3 nd gtewy host : defult vi dev enp0s3 proto sttic metric 100 Updte the network device. Replce Device-Nme with the nme of the device to updte: ip link set Device-Nme down ; \ ip link set Device-Nme nme eth0 ; ip link set eth0 up c To mintin your terminl session, perform the preceding commnds s shown. Updte the defult route. Replce Gtewy-Address with the IP ddress of the gtewy host: ip route replce defult vi Gtewy-Address dev eth0 8 Reoot the host. Prepring the mster host operting system Use this procedure to prepre RHEL/CentOS host s Control Center mster host. 1 Log in to the cndidte mster host s root, or s user with superuser privileges. 2 Disle the firewll, if necessry. This step is required for instlltion ut not for deployment. For more informtion, refer to the Control Center Plnning Guide. Determine whether the firewlld service is enled. systemctl sttus firewlld.service If the result includes Active: inctive (ded), the service is disled. Proceed to the next step. If the result includes Active: ctive (running), the service is enled. Perform the following sustep. Disle the firewlld service. systemctl stop firewlld && systemctl disle firewlld On success, the preceding commnds disply messges similr to the following exmple: rm '/etc/systemd/system/dus-org.fedorproject.firewlld1.service' rm '/etc/systemd/system/sic.trget.wnts/firewlld.service' 3 Optionl: Enle persistent storge for log files, if desired. By defult, RHEL/CentOS systems store log dt only in memory or in ring uffer in the /run/log/ journl directory. By performing this step, log dt persists nd cn e sved indefinitely, if you implement log file rottion prctices. For more informtion, refer to your operting system documenttion. mkdir -p /vr/log/journl && systemctl restrt systemd-journld 4 Disle Security-Enhnced Linux (SELinux), if instlled. 15

16 Control Center Instlltion Guide for High-Avilility Deployments Determine whether SELinux is instlled. test -f /etc/selinux/config && grep '^SELINUX=' /etc/selinux/config c If the preceding commnds return result, SELinux is instlled. Set the operting mode to disled. Open /etc/selinux/config in text editor, nd chnge the vlue of the SELINUX vrile to disled. Confirm the new setting. grep '^SELINUX=' /etc/selinux/config 5 Enle nd strt the Dnsmsq pckge. The pckge fcilittes networking mong Docker continers. systemctl enle dnsmsq && systemctl strt dnsmsq If nme resolution in your environment relies solely on entries in /etc/hosts, configure dsnmsq so tht continers cn use the file: c d e f g h i j k Open /etc/dnsmsq.conf with text editor. Locte the line tht strts with #domin-needed, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Locte the line tht strts with #ogus-priv, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Locte the line tht strts with #locl=/loclnet/, nd then mke copy of the line, immeditely elow the originl. Remove net, nd then remove the numer sign chrcter (#) from the eginning of the line. Locte the line tht strts with #domin=exmple.com, nd then mke copy of the line, immeditely elow the originl. Replce exmple.com with locl, nd then remove the numer sign chrcter (#) from the eginning of the line. Sve the file, nd then close the editor. Restrt the dnsmsq service. systemctl restrt dnsmsq 6 Add the required hostnmes nd IP ddresses of oth the primry nd the secondry node to the /etc/ hosts file. For dul-nic system, replce ech vrile nme with the vlues designted for ech node, nd replce exmple.com with the domin nme of your orgniztion: echo "Primry-Pulic-IP Primry-Pulic-Nme.exmple.com \ Primry-Pulic-Nme" >> /etc/hosts echo "Primry-Privte-IP Primry-Privte-Nme.exmple.com \ Primry-Privte-Nme" >> /etc/hosts echo "Secondry-Pulic-IP Secondry-Pulic-Nme.exmple.com \ Secondry-Pulic-Nme" >> /etc/hosts echo "Secondry-Privte-IP Secondry-Privte-Nme.exmple.com \ Secondry-Privte-Nme" >> /etc/hosts 16

17 Instlling mster host For single-nic system, replce ech vrile nme with the vlues designted for ech node, nd replce exmple.com with the domin nme of your orgniztion: echo "Primry-Pulic-IP Primry-Pulic-Nme.exmple.com \ Primry-Pulic-Nme" >> /etc/hosts echo "Secondry-Pulic-IP Secondry-Pulic-Nme.exmple.com \ Secondry-Pulic-Nme" >> /etc/hosts 7 Crete mount point for ppliction dt ckups. The defult mount point is /opt/serviced/vr/ckups. You cn chnge the defult y editing the SERVICED_BACKUPS_PATH vrile in the Control Center configurtion file. mkdir -p /opt/serviced/vr/ckups 8 Crete mount point for Control Center internl services dt. The defult mount point is /opt/serviced/vr/isvcs. You cn chnge the defult y editing the SERVICED_ISVCS_PATH vrile in the Control Center configurtion file. mkdir -p /opt/serviced/vr/isvcs 9 Remove file system signtures from the required storge res. Replce ech vrile nme with the pth of ech storge re: wipefs - Isvcs-Storge wipefs - Metdt-Storge wipefs - App-Dt-Storge 10 Reoot the host. reoot Instlling required softwre nd imges Mster host nodes need cluster-mngement pckges nd Control Center softwre to perform their roles in high-vilility deployment. Use the procedures in the following sections to downlod nd stge or instll required softwre nd Docker Engine imges. Perform the procedures for ech mster host node. Downloding repository nd imge files To perform this procedure, you need: A worksttion with internet ccess. Permission to downlod files from the File Portl - Downlod Zenoss Enterprise Softwre site. Zenoss customers cn request permission y filing ticket t the Zenoss Support site. A secure network copy progrm. Use this procedure to downlod the required files to worksttion copy the files to the hosts tht need them Perform these steps: 17

18 Control Center Instlltion Guide for High-Avilility Deployments 1 In we rowser, nvigte to the File Portl - Downlod Zenoss Enterprise Softwre site. 2 Log in with the ccount provided y Zenoss Support. 3 Downlod the self-instlling Docker imge files. instll-zenoss-serviced-isvcs:v56.run instll-zenoss-isvcs-zookeeper:v8.run 4 Downlod the Control Center RPM file. serviced x86_64.rpm 5 Downlod RHEL/CentOS repository mirror file. The downlod site provides repository mirror file for ech supported relese of RHEL/CentOS. Ech file contins the Control Center pckge nd its dependencies. To downlod the correct repository mirror file, mtch the operting system relese numer in the file nme with the version of RHEL/CentOS instlled on ll of the hosts in your Control Center cluster. yum-mirror-centos7.centos7.1-version.x86_64.rpm yum-mirror-centos7.centos7.2-version.x86_64.rpm yum-mirror-centos7.centos7.3-version.x86_64.rpm 6 Optionl: Downlod the Pcemker resource gents for Control Center. The resource gents re required only for high-vilility deployments. serviced-resource-gents x86_64.rpm 7 Use secure copy progrm to copy the files to Control Center cluster hosts. Copy ll files to the mster host or oth mster nodes (high-vilility deployments). Copy the RHEL/CentOS RPM file nd the Control Center RPM file to ll delegte hosts. Copy the Docker imge file for ZooKeeper to delegte hosts tht re ZooKeeper ensemle nodes. Instlling the repository mirror Use this procedure to instll the Zenoss repository mirror on Control Center host. The mirror contins pckges tht re required on ll Control Center cluster hosts. 1 Log in to the trget host s root, or s user with superuser privileges. 2 Move the RPM files to /tmp. 3 Optionl: Remove the existing repository mirror, if necessry. This step is not necessry during instlltions, only upgrdes. Serch for the existing repository mirror. yum list --dislerepo=* wk '/^yum-mirror/ { print $1}' Remove the mirror. Replce Old-Mirror with the nme of the Zenoss repository mirror returned in the previous sustep: yum remove -y Old-Mirror 4 Instll the repository mirror. yum instll -y /tmp/yum-mirror-*.rpm The yum commnd copies the contents of the RPM file to /opt/zenoss-repo-mirror. 18

19 Instlling mster host 5 Copy the Control Center RPM file to the mirror directory. cp /tmp/serviced rpm /opt/zenoss-repo-mirror 6 Copy the Pcemker resource gents for Control Center to the mirror directory. cp /tmp/serviced-resource-gents x86_64.rpm \ /opt/zenoss-repo-mirror 7 Optionl: Delete the RPM files. rm /tmp/yum-mirror-*.rpm /tmp/serviced-*.rpm Stging Docker imge files on the mster host Before performing this procedure, verify tht pproximtely 640MB of temporry spce is ville on the file system where /root is locted. Use this procedure to dd Docker imge files to the Control Center mster host. The files re used when Docker is fully configured. 1 Log in to the mster host s root, or s user with superuser privileges. 2 Copy or move the rchive files to /root. 3 Add execute permission to the files. chmod +x /root/*.run Downloding nd stging cluster softwre To perform this procedure, you need: An RHEL/CentOS system with internet ccess nd the sme operting system relese nd kernel s the mster host nodes. A secure network copy progrm. Use this procedure to downlod pckges for Distriuted Replicted Block Device (DRBD) nd Pcemker/ Corosync, nd to undle them for instlltion on mster host nodes. 1 Log in to comptile host tht is connected to the internet s root, or s user with superuser privileges. The host must hve the sme operting system (RHEL or CentOS) nd relese instlled, nd the sme version of the Linux kernel, s the mster host nodes. 2 Instll yum utilities, if necessry. Determine whether the yum utilities pckge is instlled. rpm -q grep yum-utils If the commnd returns result, the pckge is instlled. Proceed to the next step. If the commnd does not return result, the pckge is not instlled. Perform the following sustep. Instll the yum-utils pckge. yum instll -y yum-utils 3 Add the Enterprise Linux pckges repository (ELRepo), if necessry. 19

20 Control Center Instlltion Guide for High-Avilility Deployments Determine whether the ELRepo repository is ville. yum repolist grep elrepo c d If the commnd returns result, the repository is ville. Proceed to the next step. If the commnd does not return result, the repository is not ville. Perform the following susteps. Import the pulic key for the repository. rpm --import Add the repository to the downlod host. rpm -Uvh \ Clen nd updte the yum cches. yum clen ll && yum mkecche fst 4 Downlod the required pckges nd their dependencies, nd then crete tr rchive of the pckge files. Crete temporry directory for the pckges. c d mkdir /tmp/downlods Downlod the DRBD pckges to the temporry directory. repotrck - x86_64 -r elrepo -p /tmp/downlods kmod-drd84 Downlod the Corosync/Pcemker pckges to the temporry directory. repotrck - x86_64 -p /tmp/downlods pcs Crete tr rchive of the temporry directory. cd /tmp && tr czf./downlods.tgz./downlods 5 Use secure copy progrm to copy the pckges rchive to the /tmp directory of ech mster host node. Configuring privte mster NTP server Control Center requires common time source. The following procedure configures privte mster NTP server to synchronize the system clocks of ll hosts in Control Center cluster. Note VMwre vsphere guest systems cn synchronize their system clocks with the host system. If tht feture is enled, it must e disled to configure privte mster NTP server. For more informtion, refer to the VMwre documenttion for your version of vsphere. Instlling nd configuring n NTP mster server Use this procedure to configure n NTP mster server on mster host node. 20

21 Instlling mster host Note On VMwre vsphere guests, disle time synchroniztion etween guest nd host operting systems efore performing this procedure. 1 Log in to the mster host node s root, or s user with superuser privileges. 2 Instll the NTP pckge. yum --dislerepo=\* --enlerepo=zenoss-mirror instll -y ntp 3 Crete ckup of the NTP configurtion file. cp -p /etc/ntp.conf /etc/ntp.conf.orig 4 Edit the NTP configurtion file./ Open /etc/ntp.conf with text editor. Replce ll of the lines in the file with the following lines: # Use the locl clock server prefer fudge strtum 10 driftfile /vr/li/ntp/drift rodcstdely # Give loclhost full ccess rights restrict c # Grnt ccess to client hosts restrict ADDRESS_RANGE msk NETMASK nomodify notrp Replce ADDRESS_RANGE with the rnge of IPv4 network ddresses tht re llowed to query this NTP server. For exmple, the following IP ddresses re ssigned to the hosts in n Control Center cluster: For the preceding ddresses, the vlue for ADDRESS_RANGE is d Replce NETMASK with the IPv4 network msk tht corresponds with the ddress rnge. For exmple, vlid network msk for is e Sve the file nd exit the editor. 5 Enle nd strt the NTP demon. Enle the ntpd demon. systemctl enle ntpd Configure ntpd to strt when the system strts. Currently, n unresolved issue ssocited with NTP prevents ntpd from restrting correctly fter reoot, nd the following commnds provide workround to ensure tht it does. echo "systemctl strt ntpd" >> /etc/rc.d/rc.locl chmod +x /etc/rc.d/rc.locl 21

22 Control Center Instlltion Guide for High-Avilility Deployments c Strt ntpd. systemctl strt ntpd Instlling Docker Engine Use this procedure to instll Docker Engine. 1 Log in to the host s root, or s user with superuser privileges. 2 Instll Docker Engine from the locl repository mirror. Clen the yum cche nd updte repository metdt. yum clen ll && yum mkecche fst Instll Docker Engine. yum instll -y --dislerepo=\* --enlerepo=zenoss-mirror \ docker-engine c Enle utomtic strtup. systemctl enle docker Instlling Control Center Use this procedure to instll Control Center. 1 Log in to the mster host s root, or s user with superuser privileges. 2 Instll Control Center from the locl repository mirror. Clen the yum cche nd updte repository metdt. yum clen ll && yum mkecche fst Instll Control Center. yum --dislerepo=\* --enlerepo=zenoss-mirror instll -y \ /opt/zenoss-repo-mirror/serviced x86_64.rpm 3 Disle utomtic strtup of Control Center. The cluster mngement softwre controls the serviced service. systemctl disle serviced 4 Mke ckup copy of the Control Center configurtion file. Mke copy of /etc/defult/serviced. cp /etc/defult/serviced /etc/defult/serviced orig Set the ckup file permissions to red-only. chmod 0440 /etc/defult/serviced orig 5 Add drop-in file for the NFS service. 22

23 Instlling mster host This step is workround for n n unresolved issue. Crete directory for the drop-in file. mkdir -p /etc/systemd/system/nfs-server.service.d Crete the drop-in file. ct <<EOF > /etc/systemd/system/nfs-server.service.d/nfs-server.conf [Unit] Requires= Requires= network.trget proc-fs-nfsd.mount rpcind.service Requires= nfs-mountd.service EOF c Relod the systemd mnger configurtion. systemctl demon-relod Configuring Docker Engine nd loding imges Use this procedure to configure Docker Engine nd lod imges in to the locl repository. 1 Log in to the mster host s root, or s user with superuser privileges. 2 Crete symolic link for the Docker temporry directory. Docker uses its temporry directory to spool imges. The defult directory is /vr/li/docker/tmp. The following commnd specifies the sme directory tht Control Center uses, /tmp. You cn specify ny directory tht hs minimum of 10GB of unused spce. Crete the docker directory in /vr/li. mkdir /vr/li/docker Crete the link to /tmp. ln -s /tmp /vr/li/docker/tmp 3 Crete systemd drop-in file for Docker Engine. Crete the override directory. mkdir -p /etc/systemd/system/docker.service.d Crete the unit drop-in file. ct <<EOF > /etc/systemd/system/docker.service.d/docker.conf [Service] TimeoutSec=300 EnvironmentFile=-/etc/sysconfig/docker ExecStrt= ExecStrt=/usr/in/dockerd \$OPTIONS TsksMx=infinity EOF c Relod the systemd mnger configurtion. systemctl demon-relod 23

24 Control Center Instlltion Guide for High-Avilility Deployments 4 Crete n LVM thin pool for Docker dt. For more informtion out the serviced-storge commnd, see serviced-storge on pge 92. To use n entire lock device or prtition for the thin pool, replce Device-Pth with the device pth: serviced-storge crete-thin-pool docker Device-Pth To use 50GB of n LVM volume group for the thin pool, replce Volume-Group with the nme of n LVM volume group: serviced-storge crete-thin-pool --size=50g docker Volume-Group On success, the result is the device mpper nme of the thin pool, which lwys strts with /dev/mpper. 5 Configure nd strt the Docker service. Crete vrile for the nme of the Docker thin pool. Replce Thin-Pool-Device with the nme of the thin pool device creted in the previous step: c d mypool="thin-pool-device" Crete vriles for dding rguments to the Docker configurtion file. The --exec-opt rgument is workround for Docker issue on RHEL/CentOS 7.x systems. mydriver="--storge-driver devicempper" mylog="--log-level=error" myfix="--exec-opt ntive.cgroupdriver=cgroupfs" mymount="--storge-opt dm.mountopt=discrd" myflg="--storge-opt dm.thinpooldev=$mypool" Add the rguments to the Docker configurtion file. echo 'OPTIONS="'$myLog $mydriver $myfix $mymount $myflg'"' \ >> /etc/sysconfig/docker Strt or restrt Docker. systemctl restrt docker The strtup my tke up to minute, nd my fil. If strtup fils, repet the restrt commnd. 6 Configure nme resolution in continers. Ech time it strts, docker selects n IPv4 sunet for its virtul Ethernet ridge. The selection cn chnge; this step ensures consistency. Identify the IPv4 sunet nd netmsk docker hs selected for its virtul Ethernet ridge. c ip ddr show docker0 grep inet Open /etc/sysconfig/docker in text editor. Add the following flgs to the end of the OPTIONS declrtion. Replce Bridge-Sunet with the IPv4 sunet docker selected for its virtul ridge: --dns=bridge-sunet --ip=bridge-sunet/16 24

25 Instlling mster host For exmple, if the ridge sunet is , dd the following flgs: --dns= ip= /16 Note Use spce chrcter ( ) to seprte flgs, nd mke sure the doule quote chrcter (") delimits the declrtion of OPTIONS. d Restrt the Docker service. systemctl restrt docker 7 Import the Control Center imges into the locl Docker Engine repository. The imges re contined in the self-extrcting rchive files tht re stged in /root. Chnge directory to /root. cd /root Extrct the imges. for imge in instll-zenoss-*.run do /in/echo -n "$imge: "./$imge done c Imge extrction egins when you press y. If you press y nd then Enter, the current imge is extrcted, ut the next one is not. Optionl: Delete the rchive files, if desired. rm -i./instll-zenoss-*.run 8 Stop nd disle the Docker Engine service. The cluster mngement softwre controls the Docker Engine service. systemctl stop docker && systemctl disle docker 25

26 Control Center Instlltion Guide for High-Avilility Deployments Configuring Control Center on mster host 3 nodes This chpter includes the procedures for configuring Control Center on mster host nodes, nd descries the configurtion options tht pply to mster hosts. Mny configurtion choices depend on ppliction requirements. Plese review your ppliction documenttion efore configuring Control Center. Note Perform the procedures in this chpter on the primry node nd the secondry node. This chpter includes synopses of the configurtion vriles tht ffect the mster host. For more informtion out vrile, see Control Center configurtion vriles on pge 96. Control Center mintennce scripts on the mster host The scripts in the following list re instlled when Control Center is instlled, nd re strted either dily or weekly y ncron. /etc/cron.dily/serviced This script invokes logrotte dily, to mnge the /vr/log/serviced.ccess.log file. This script is required on the mster host nd on ll delegte hosts. /etc/cron.weekly/serviced-fstrim This script invokes fstrim weekly, to reclim unused locks in the ppliction dt thin pool. The life spn of solid-stte drive (SSD) degrdes when fstrim is run too frequently. If the lock storge of the ppliction dt thin pool is n SSD, you cn reduce the frequency t which this script is invoked, s long s the thin pool never runs out of free spce. An identicl copy of this script is locted in /opt/ serviced/in. This script is required on the mster host only. /etc/cron.weekly/serviced-zenossdpck This script invokes serviced commnd weekly, which in turn invokes the dtse mintennce script for Zenoss ppliction. If the ppliction is not instlled or is offline, the commnd fils. This script is required on the mster host only. User ccess control Control Center provides rowser interfce nd commnd-line interfce. 26

27 Configuring Control Center on mster host nodes To gin ccess to the Control Center rowser interfce, users must hve login ccounts on the Control Center mster host. In ddition, users must e memers of the Control Center dministrtive group, which y defult is the system group, wheel. To enhnce security, you my chnge the dministrtive group from wheel to ny non-system group. To use the Control Center commnd-line interfce (CLI) on Control Center cluster host, user must hve login ccount on the host, nd the ccount must e memer of the serviced group. Pluggle Authentiction Modules (PAM) is supported nd recommended for enling ccess to oth the rowser interfce nd the commnd-line interfce. Adding users to the defult dministrtive group Use this procedure to dd users to the defult dministrtive group of Control Center, wheel. Performing this procedure enles users to gin ccess to the Control Center rowser interfce. Note Perform this procedure or the next procedure, ut not oth. 1 Log in to the host s root, or s user with superuser privileges. 2 Add user to the system group. Replce User with the nme of login ccount on the mster host. usermod -G wheel User Repet the preceding commnd for ech user to dd. Configuring regulr group s the Control Center dministrtive group Use this procedure to chnge the defult dministrtive group of Control Center from wheel to non-system group. The following Control Center vriles re used in this procedure: SERVICED_ADMIN_GROUP Defult: wheel The nme of the Linux group on the serviced mster host whose memers re uthorized to use the serviced rowser interfce. You my replce the defult group with group tht does not hve superuser privileges. SERVICED_ALLOW_ROOT_LOGIN Defult: 1 (true) Determines whether the root user ccount on the serviced mster host my e used to gin ccess to the serviced rowser interfce. Note Perform this procedure or the previous procedure, ut not oth. 1 Log in to the host s root, or s user with superuser privileges. 2 Crete vrile for the group to designte s the dministrtive group. In this exmple, the group is serviced. You my crete nd use different group. mygroup=serviced 3 Add one or more existing users to the group. 27

28 Control Center Instlltion Guide for High-Avilility Deployments Replce User with the nme of login ccount on the host: usermod -G $mygroup User Repet the preceding commnd for ech user to dd. 4 Specify the new dministrtive group in the serviced configurtion file. Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_ADMIN_GROUP vrile, nd then mke copy of the line, immeditely elow the originl. c Remove the numer sign chrcter (#) from the eginning of the line. d Chnge the vlue from wheel to the nme of the group you chose erlier. e Sve the file, nd then close the editor. 5 Optionl: Prevent the root user from gining ccess to the Control Center rowser interfce, if desired. Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_ALLOW_ROOT_LOGIN vrile, nd then mke copy of the line, immeditely elow the originl. c Remove the numer sign chrcter (#) from the eginning of the line. d Chnge the vlue from 1 to 0. e Sve the file, nd then close the editor. Enling use of the commnd-line interfce Use this procedure to enle user to perform dministrtive tsks with the Control Center commnd-line interfce. 1 Log in to the host s root, or s user with superuser privileges. 2 Add user to the serviced group. Replce User with the nme of login ccount on the host. usermod -G serviced User Repet the preceding commnd for ech user to dd. Configuring the se size device for tennt dt storge Use this procedure to configure the se size of virtul storge devices for tennts in the ppliction dt thin pool. The se size is used ech time tennt device is creted. In prticulr, the first time serviced strts, it cretes the se size device nd then cretes tennt device from the se size device. The following Control Center configurtion vrile is used to specify the se size device: SERVICED_DM_BASESIZE Defult: 100G The se size of virtul storge devices for tennts in the ppliction dt thin pool, in gigytes. The units symol (G) is required. This vrile is used when serviced strts for the first time, to set the initil size of tennt devices, nd when ckup is restored, to set the size of the restored tennt device. The se size device is sprse device tht occupies t most 1MB of spce in the ppliction dt thin pool; its size hs no immedite prcticl impct. However, the ppliction dt thin pool should hve enough spce for twice the size of ech tennt device it supports, to store oth the dt itself nd snpshots of the dt. Since the ppliction dt thin pool is n LVM logicl volume, its size cn e incresed t ny time. 28

29 Configuring Control Center on mster host nodes Likewise, the size of tennt device cn e incresed, s long s the ville spce in the thin pool cn support the lrger tennt device plus snpshots. Perform these steps: 1 Log in to the mster host s root, or s user with superuser privileges. 2 Identify the size of the thin pool for ppliction dt. The size is required to set n ccurte vlue for the SERVICED_DM_BASESIZE vrile. lvs --options=lv_nme,lv_size grep serviced-pool 3 Edit storge vriles in the Control Center configurtion file. c d Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_DM_BASESIZE vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Chnge the vlue, if necessry. Replce Fifty-Percent with the vlue tht is less thn or equl to 50% of the size of the thin pool for ppliction dt. Include the symol for gigytes, G: SERVICED_DM_BASESIZE=Fifty-PercentG e Sve the file, nd then close the editor. 4 Verify the settings. grep -E '^\*SERVICED' /etc/defult/serviced Setting the host role to mster Use this procedure to configure host s the mster host. The following configurtion vrile ssigns the host role: SERVICED_MASTER Defult: 1 (true) Assigns the role of serviced instnce, either mster or delegte. The mster runs the ppliction services scheduler nd other internl services. Delegtes run the ppliction services ssigned to the resource pool to which they elong. Only one serviced instnce cn e the mster; ll other instnces must e delegtes. The defult vlue ssigns the mster role. To ssign the delegte role, set the vlue to 0 (flse). This vrile must e explicitly set on ll Control Center cluster hosts. Perform these steps: 1 Log in to the host s root, or s user with superuser privileges. 2 Edit the Control Center configurtion file. c d Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_MASTER vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Sve the file, nd then close the editor. 29

30 Control Center Instlltion Guide for High-Avilility Deployments 3 Verify the settings. grep -E '^\*SERVICED' /etc/defult/serviced Configuring offline use Use this procedure to configure mster host nodes to send their responses to the virtul IP ddress of the highvilility cluster. The following configurtion vrile specifies the IP ddress: SERVICED_OUTBOUND_IP Defult: (none) The defult strtup routines of serviced include ttempting to ping google.com. When vlue is set for this vrile, serviced does not ttempt the ping nd ssumes it does not hve internet ccess. Use this vrile to specify the IP ddress of network interfce other thn the defult, or to prevent serviced from ssuming it hs internet ccess. Note Setting the Docker HTTP_PROXY or HTTPS_PROXY environment vriles prevents ccess to the IP ddress defined with this vrile. To enle ccess, unset the Docker vriles, nd then reoot the host. Perform these steps: 1 Log in to the host s root, or s user with superuser privileges. 2 Edit the Control Center configurtion file. Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_OUTBOUND_IP vrile, nd then mke copy of the line, immeditely elow the originl. c Remove the numer sign chrcter (#) from the eginning of the line. d Chnge the vlue to the virtul IP ddress of the high-vilility cluster (HA-Virtul-IP). e Sve the file, nd then close the editor. 3 Verify the settings. grep -E '^\*SERVICED' /etc/defult/serviced Configuring endpoints Use this procedure to customize the internl services endpoints of Control Center. The following configurtion vriles specify the endpoints: SERVICED_ZK Defult: (none) The list of endpoints in the serviced ZooKeeper ensemle, seprted y the comm chrcter (,). Ech endpoint identifies n ensemle node. SERVICED_DOCKER_REGISTRY Defult: {{SERVICED_MASTER_IP}}:5000 The endpoint of the serviced Docker registry host. On delegte hosts, replce {{SERVICED_MASTER_IP}} with the IP ddress or hostnme of the registry host, which y defult is the serviced mster host. On the mster host, this vrile is ignored. The vlue used to replce {{SERVICED_MASTER_IP}} in this vrile must mtch the vlue of the -- insecure-registry flg in the /etc/sysconfig/docker file. 30

31 Configuring Control Center on mster host nodes SERVICED_ENDPOINT Defult: {{SERVICED_MASTER_IP}}:4979 The endpoint of the serviced RPC server. Replce {{SERVICED_MASTER_IP}} with the IP ddress or hostnme of the serviced mster host. The port numer of this endpoint must mtch the vlue of the SERVICED_RPC_PORT vrile defined on the serviced mster host. SERVICED_LOG_ADDRESS Defult: {{SERVICED_MASTER_IP}}:5042 The endpoint of the logstsh service. Replce {{SERVICED_MASTER_IP}} with the IP ddress or hostnme of the serviced mster host. SERVICED_LOGSTASH_ES Defult: {{SERVICED_MASTER_IP}}:9100 The endpoint of the Elsticserch service for logstsh. On delegte hosts, replce {{SERVICED_MASTER_IP}} with the IP ddress or hostnme of the Elsticserch host, which y defult is the serviced mster host. SERVICED_STATS_PORT Defult: {{SERVICED_MASTER_IP}}:8443 The endpoint of the serviced metrics consumer service. Replce {{SERVICED_MASTER_IP}} with the IP ddress or hostnme of the serviced mster host. Perform these steps: 1 Log in to the host s root, or s user with superuser privileges. 2 Edit the Control Center configurtion file. Open /etc/defult/serviced in text editor. Glolly replce {{SERVICED_MASTER_IP}} with the virtul IP ddress of the high-vilility cluster (HA-Virtul-IP). Note Remove the numer sign chrcter (#) from the eginning of ech vrile declrtion tht includes the IP ddress. c Sve the file, nd then close the editor. 3 Verify the settings. grep -E '^\*SERVICED' /etc/defult/serviced Mster host configurtion vriles The tles in this section provide n overview of the serviced configurtion vriles tht pply to the Control Center mster host. Set these vriles s required for your environment or pplictions. Best prctices for configurtion files The Control Center configurtion file, /etc/defult/serviced, contins Bsh environment vriles tht re red y the serviced demon strtup script. The following list descries recommended est prctices for its use nd mintennce: 1 When in dout, mke ckup. Before editing, mking ckup copy of the configurtion file is lwys the sfest choice. 31