Control Center Installation Guide

Control Center Instlltion Guide Relese 1.5.0 Zenoss, Inc. www.zenoss.com

Control Center Instlltion Guide Copyright 2017 Zenoss, Inc. All rights reserved. Zenoss, Own IT, nd the Zenoss logo re trdemrks or registered trdemrks of Zenoss, Inc., in the United Sttes nd other countries. All other trdemrks, logos, nd service mrks re the property of Zenoss or other third prties. Use of these mrks is prohiited without the express written consent of Zenoss, Inc., or the third-prty owner. Linux is registered trdemrk of Linus Torvlds. All other compnies nd products mentioned re trdemrks nd property of their respective owners. Prt Numer: 1920.17.349 Zenoss, Inc. 11305 Four Points Drive Bldg 1 - Suite 300 Austin, Texs 78726 2

Contents Aout this guide...5 Supported clients nd rowsers...5 Relted pulictions...5 Chnge history... 6 Chpter 1: Downloding nd stging required files...8 Downloding Control Center files... 8 Instlling the repository mirror...9 Stging Docker imge files... 11 Stging Docker imge file on ZooKeeper ensemle nodes... 11 Chpter 2: Instlling mster host... 12 Verifying cndidte host resources... 12 Mster host storge requirements...14 Prepring the mster host operting system...16 Instlling Docker CE nd Control Center...18 Configuring Docker... 19 Loding imge files... 20 Creting the ppliction dt thin pool... 21 Chpter 3: Configuring nd strting the mster host... 23 Control Center mintennce scripts on the mster host...23 User ccess control... 24 Configuring the se size device for tennt dt storge... 26 Setting the host role to mster... 26 Optionl: Chnging the locl Docker registry endpoint... 27 Optionl: Configuring offline use... 28 Mster host configurtion vriles... 29 Universl configurtion vriles...32 Strting Control Center for the first time...34 Adding the mster host to resource pool... 35 Chpter 4: Instlling delegte hosts...36 Verifying cndidte host resources... 36 Delegte host storge requirements...38 Prepring delegte host...38 Instlling Docker CE nd Control Center...40 Configuring NFS 4.0... 41 Configuring Docker... 41 Chpter 5: Configuring nd strting delegte hosts...44 Control Center mintennce scripts on delegte hosts...44 Enling use of the commnd-line interfce...44 Setting the host role to delegte...45 Chnging the locl Docker registry endpoint... 45 3

Setting internl services endpoints... 46 Optionl: Configuring offline use... 47 Delegte host configurtion vriles... 48 Universl configurtion vriles...50 Strting Control Center...51 Delegte host uthentiction...52 Chpter 6: Configuring ZooKeeper ensemle... 55 ZooKeeper nd Control Center... 55 Understnding the configurtion process... 55 Configuring the mster host s ZooKeeper node...57 Configuring delegte host A s ZooKeeper node...58 Configuring delegte host B s ZooKeeper node... 60 Importing the ZooKeeper imge for Docker...61 Strting ZooKeeper ensemle... 62 Updting delegte hosts...63 Appendix A: Strting nd stopping Control Center deployments... 64 Stopping Control Center (single-host deployment)...64 Strting Control Center (single-host deployment)... 65 Stopping Control Center (multi-host deployment)...66 Strting Control Center (multi-host deployment)... 69 Appendix B: Storge mngement utility...71 serviced-storge...71 Appendix C: Control Center configurtion vriles... 75 Best prctices for configurtion files... 75 Control Center configurtion file... 75 Appendix D: Configuring privte mster NTP server... 86 Configuring n NTP mster server... 86 Configuring NTP clients...87 Appendix E: Resolving pckge dependency conflicts... 89 Resolving device mpper dependency conflicts... 89 Resolving other dependency conflicts...90 4

Aout this guide Aout this guide Control Center Instlltion Guide provides detiled procedures for instlling nd configuring Control Center deployment. Before using this guide, crefully review the Control Center Plnning Guide. Zenoss customers: This guide does not include procedures for instlling high-vilility deployment. For more informtion, refer to the Control Center Instlltion Guide for High-Avilility Deployments. Supported clients nd rowsers The following tle identifies the supported comintions of client operting systems nd we rowsers. Client OS Windows 7, 10 mcos 10.12.3, 10.13 Uuntu 14.04 LTS Tested rowsers Internet Explorer 11 * Firefox 56 nd lter Chrome 61 nd lter Firefox 56 nd lter Chrome 61 nd lter Firefox 56 nd lter Chrome 61 nd lter Relted pulictions Title Control Center Relese Notes Control Center Plnning Guide Control Center Instlltion Guide Control Center Instlltion Guide for High-Avilility Deployments Control Center Reference Guide Control Center Upgrde Guide Control Center Upgrde Guide for High- Avilility Deployments Description Descries known issues, fixed issues, nd lte-reking informtion not included in other pulictions. Provides oth generl nd specific informtion out prepring to deploy Control Center. Provides detiled procedures for instlling nd configuring Control Center. Provides detiled procedures for instlling nd configuring Control Center in high-vilility deployment. Provides informtion nd procedures for mnging Control Center. This informtion is lso ville s online help in the Control Center rowser interfce. Provides detiled procedures for updting Control Center deployment to the ltest relese. Provides detiled procedures for updting high-vilility deployment of Control Center to the ltest relese. * Enterprise mode only; comptiility mode is not tested. 5

Control Center Instlltion Guide Documenttion feedck To provide feedck out this document, or to report n error or omission, plese send n emil to docs@controlcenter.io. In the emil, plese include the document title (Control Center Instlltion Guide) nd prt numer (1920.17.349) nd s much informtion s possile out the context of your feedck. Chnge history The following list ssocites document prt numers nd the importnt chnges to this guide since the previous relese. Some of the chnges involve fetures or content, ut others do not. For informtion out new or chnged fetures, refer to the Control Center Relese Notes. 1920.17.349 (1.5.0) Specify Docker version in yum instll commnd. Reinstte steps to disle firewll. 1920.17.331 (1.5.0) Replce Lepfile.net with delivery.zenoss.com. Add steps for importing the Zenoss GPG key. 1920.17.311 (1.5.0) Replce Docker CE 17.03.1 with 17.09.0. Replce RHEL/CentOS 7.1 with 7.4. 1320.17.268 Updte relese numer (1.4.1). 1320.17.242 Correct previous modifiction for configuring Docker. The --insecure-registry flg must e set when the vlue of SERVICED_DOCKER_REGISTRY is not loclhost:5000. 1320.17.187 Modify the yum commnd used to instll Docker CE 17.03.1. 1320.17.171 Updte relese numer (1.3.3). Add Docker CE 17.03.1. Add ZooKeeper vriles for tuning TCP/IP communictions with resource pools. Remove step for disling SELinux. 1320.17.122 Updte relese numer (1.3.2). 1320.17.100 Updte relese numer (1.3.1). Modify the step for configuring Docker on mster hosts. The --insecure-registry flg is only needed on delegte hosts. 1320.17.076 Updte relese numer (1.3.0). Add clrifiction tht offline instlltions must e performed completely offline. Add procedure for dding delegtes when using network ddress trnsltion (NAT). 1320.17.59 Updte relese numer (1.2.3). Remove TLS_RSA_WITH_RC4_128_SHA from list of ciphers ssocited with SERVICED_TLS_CIPHERS. 6

Aout this guide 1320.17.024 Updte relese numer (1.2.2). 1320.16.351 Minor chnges to synchronize with the Upgrde Guide. 1320.16.350 Add steps to downlod nd use the serviced RPM file (Zenoss customers only). Add procedures for importing the ZooKeeper imge file on offline nodes (Zenoss customers only). Updte relese numer (1.2.1). 1320.16.327 Add new est prctice guideline for mnging configurtion files. 1320.16.319 Initil relese (1.2.0). 7

Control Center Instlltion Guide Downloding nd stging required files 1 This chpter descries how to downlod nd instll or stge Control Center softwre nd its operting system dependencies. The procedures in this chpter re required to perform n instlltion. The following tle identifies where to perform ech procedure in this chpter. Procedure Downloding Control Center files on pge 8 Instlling the repository mirror on pge 9 Stging Docker imge files on pge 11 Stging Docker imge file on ZooKeeper ensemle nodes on pge 11 Where to perform A worksttion with internet ccess All Control Center hosts The Control Center mster host Delegte hosts tht re ZooKeeper ensemle nodes Downloding Control Center files To perform this procedure, you need: A worksttion with internet ccess. Zenoss Resource Mnger users: Permission to downlod files from delivery.zenoss.com. Customers cn request permission y filing ticket t the Zenoss Support site. Zenoss Core users: An ccount on the Zenoss Community site. A secure network copy progrm. Use this procedure to downlod the required files to worksttion copy the files to the hosts tht need them Perform these steps: 1 In we rowser, nvigte to the downlod site, nd then log in. Zenoss Resource Mnger users: delivery.zenoss.com Zenoss Core users: Zenoss Community 2 Downlod the self-instlling Docker imge files. instll-zenoss-serviced-isvcs-v61.run 8

Downloding nd stging required files instll-zenoss-isvcs-zookeeper-v10.run 3 Downlod the Control Center RPM file. serviced-1.5.0-1.x86_64.rpm 4 Identify the operting system relese on Control Center hosts. Enter the following commnd on ech Control Center host in your deployment, if necessry. All Control Center hosts should e running the sme operting system relese nd kernel. ct /etc/redht-relese 5 Downlod the RHEL/CentOS repository mirror file for your deployment. The downlod site provides repository mirror file for ech tested relese of RHEL/CentOS. Ech mirror file contins the relese-specific pckges tht Control Center requires. yum-mirror-centos7.2-1511-serviced-1.5.0.x86_64.rpm yum-mirror-centos7.3-1611-serviced-1.5.0.x86_64.rpm yum-mirror-centos7.4-1708-serviced-1.5.0.x86_64.rpm 6 Optionl: Downlod the Zenoss GNU Privcy Gurd (GPG) key, if desired. You cn use the Zenoss GPG key to verify Zenoss RPM files nd the yum metdt of the repository mirror. Downlod the key. curl --loction -o /tmp/tmp.html \ 'http://keys.gnupg.net/pks/lookup?op=get&serch=0xed0a5fd2aa5a1ad7' Determine whether the downlod succeeded. grep -Ec '^\-\-\-\-\-BEGIN PGP' /tmp/tmp.html c If the result is 0, return to the previous sustep. If the result is 1, proceed to the next sustep. Extrct the key. wk '/^-----BEGIN PGP.*$/,/^-----END PGP.*$/' \ /tmp/tmp.html >./RPM-GPG-KEY-Zenoss 7 Use secure copy progrm to copy the files to Control Center hosts. Copy ll files to the mster host. Copy the RHEL/CentOS RPM file, the Control Center RPM file, nd the Zenoss GPG key file to ll delegte hosts. Copy the Docker imge file for ZooKeeper to delegte hosts tht re ZooKeeper ensemle nodes. Instlling the repository mirror Use this procedure to instll the Zenoss repository mirror on Control Center host. The mirror contins pckges tht re required on ll Control Center hosts. 1 Log in to the trget host s root, or s user with superuser privileges. 2 Move the RPM files nd the Zenoss GPG key file to /tmp. 3 Instll the repository mirror. yum instll /tmp/yum-mirror-*.rpm 9

Control Center Instlltion Guide The yum commnd copies the contents of the RPM file to /opt/zenoss-repo-mirror. 4 Optionl: Instll the Zenoss GPG key, nd then test the pckge files, if desired. Move the Zenoss GPG key to the mirror directory. c mv /tmp/rpm-gpg-key-zenoss /opt/zenoss-repo-mirror Instll the key. rpm --import /opt/zenoss-repo-mirror/rpm-gpg-key-zenoss Test the repository mirror pckge file. rpm -K /tmp/yum-mirror-*.rpm On success, the result includes the file nme nd the following informtion: d (sh1) ds sh1 md5 gpg OK Test the Control Center pckge file. rpm -K /tmp/serviced-1.5.0-1.x86_64.rpm 5 Optionl: Updte the configurtion file of the Zenoss repository mirror to enle GPG key verifiction, if desired. Open the repository mirror configurtion file (/etc/yum.repos.d/zenoss-mirror.repo) with text editor, nd then dd the following lines to the end of the file. c repo_gpgcheck=1 gpgkey=file:///opt/zenoss-repo-mirror/rpm-gpg-key-zenoss Sve the file, nd then close the editor. Updte the yum metdt cche. yum mkecche fst The cche updte process includes the following prompt: Retrieving key from file:///opt/zenoss-repo-mirror/rpm-gpg-key- Zenoss Importing GPG key 0xAA5A1AD7: Userid : "Zenoss, Inc. <dev@zenoss.com>" Fingerprint: f31f fd84 623 3d5 981d 728 ed0 5fd2 5 1d7 From : /opt/zenoss-repo-mirror/rpm-gpg-key-zenoss Is this ok [y/n]: Enter y. 6 Move the Control Center pckge file to the mirror directory. mv /tmp/serviced-1.5.0-1.x86_64.rpm /opt/zenoss-repo-mirror 7 Optionl: Delete the mirror pckge file, if desired. rm /tmp/yum-mirror-*.rpm 10

Downloding nd stging required files Stging Docker imge files Before performing this procedure, verify tht pproximtely 640MB of temporry spce is ville on the file system where /root is locted. Use this procedure to copy Docker imge files to Control Center host. The files re used when Docker is fully configured. 1 Log in to the mster host s root, or s user with superuser privileges. 2 Copy or move the rchive files to /root. 3 Add execute permission to the files. chmod +x /root/*.run Stging Docker imge file on ZooKeeper ensemle nodes Before performing this procedure, verify tht pproximtely 170MB of temporry spce is ville on the file system where /root is locted. Use this procedure to dd Docker imge file to the Control Center delegte hosts tht re ZooKeeper ensemle nodes. Delegte hosts tht re not ZooKeeper ensemle nodes do not need the file. 1 Log in to delegte host s root, or s user with superuser privileges. 2 Copy or move the instll-zenoss-isvcs-zookeeper-v10.run file to /root. 3 Add execute permission to the file. chmod +x /root/*.run 11

Control Center Instlltion Guide Instlling mster host 2 This chpter descries how to instll Control Center on Red Ht Enterprise Linux (RHEL) or CentOS host. The cndidte host must hve the CPU, RAM, nd storge resources required to serve s the Control Center mster host. For more informtion out mster host requirements, refer to the Control Center Plnning Guide. Verifying cndidte host resources Use this procedure to determine whether the hrdwre resources nd instlled operting system of host re sufficient to serve s Control Center mster host. 1 Log in to the cndidte host s root, or s user with superuser privileges. 2 Verify tht the host implements the 64-it version of the x86 instruction set. unme -m If the output is x86_64, the rchitecture is 64-it. Proceed to the next step If the output is i386/i486/i586/i686, the rchitecture is 32-it. Stop this procedure nd select different host. 3 Verify tht the host hs dequte storge spce for Docker temporry files nd udit logging. Disply the mount of spce ville in /tmp. Docker requires 10GB of storge for temporry files, nd the instlltion process includes instructions to link the Docker temporry directory to /tmp. df -h /tmp Typiclly, /tmp is mounted on the root filesystem, /. Disply the mount of spce ville in /vr/log. By defult, Control Center requires 10GB of spce for udit logging in /vr/log. The mount of spce required for udit logs, nd their loction, is configurle. For more informtion, refer to the Control Center Reference Guide. df -h /vr/log Like /tmp, /vr/log is typiclly mounted on the root filesystem, /. 12

Instlling mster host If the result does not include minimum of 10GB of spce for Docker temporry files nd 10GB for udit logging, stop this procedure nd select different host. 4 Determine whether the ville memory nd swp is sufficient. Disply the ville memory. free -h Compre the ville memory nd swp spce with the mount required for mster host in your deployment. For more informtion, refer to the Control Center Plnning Guide. If the result does not meet minimum requirements, stop this procedure nd select different host. 5 Determine whether the CPU resources re sufficient. Disply the totl numer of CPU cores. ct /proc/cpuinfo grep -Ec '^core id' Compre the ville resources with the requirements for Control Center mster host. For more informtion, refer to the Control Center Plnning Guide. 6 Determine whether the CPU resources support the AES instruction set. ct /proc/cpuinfo grep -Ec '^flgs.*es' For optiml performnce, the result of the preceding commnds must mtch the totl numer of CPU resources ville on the host. If the result is 0, performnce is severely degrded. If the result is 0 nd the cndidte host is virtul mchine, the mnging hypervisor my e configured in Hyper-V comptiility mode. Check the setting nd disle it, if possile, or select different host. 7 Determine whether the instlled operting system relese is one of the releses tht hs een tested with Control Center. ct /etc/redht-relese If the result includes 7.2, 7.3, or 7.4 proceed to the next step. If the result does not include 7.2, 7.3, or 7.4, select different host, nd then strt this procedure gin. 8 Ensure the host hs persistent numeric ID. Skip this step if you re instlling single-host deployment. Ech Control Center host must hve unique host ID, nd the ID must e persistent (not chnge when the host reoots). test -f /etc/hostid genhostid ; hostid Record the ID for comprison with other Control Center hosts. 9 Verify tht nme resolution works on this host. hostnme -i If the result is not vlid IPv4 ddress, dd n entry for the host to the network nmeserver, or to /etc/ hosts. 10 Add n entry to /etc/hosts for loclhost, if necessry. 13

Control Center Instlltion Guide Determine whether 127.0.0.1 is mpped to loclhost. grep 127.0.0.1 /etc/hosts grep loclhost If the preceding commnds return no result, perform the following sustep. Add n entry to /etc/hosts for loclhost. echo "127.0.0.1 loclhost" >> /etc/hosts 11 Updte the Linux kernel, if necessry. Determine which kernel version is instlled. unme -r If the result is lower thn 3.10.0-327.22.2.el7.x86_64, perform the following sustep. Updte the kernel, nd then restrt the host. The following commnds require internet ccess or locl mirror of operting system pckges. yum mkecche fst && yum updte kernel && reoot Mster host storge requirements This section provides quick reference for the minimum storge requirements of Control Center mster host. These requirements re in ddition to the storge required for the host's operting system. For more informtion, refer to the Control Center Plnning Guide. Control Center hosts need either unformtted lock storge devices or prtitions, or free spce in one or more LVM volume groups. Enter the following commnd to disply informtion out lock storge: lslk -p --output=name,size,type,fstype,mountpoint Enter the following commnd to disply informtion out LVM volume groups: vgdisply Compre the output with the informtion in the following tle. Purpose Minimum size Description Docker dt 50GB (required) Locl, high-performnce storge. Control Center internl services dt 50GB (required) Locl, high-performnce storge formtted s n XFS filesystem. Appliction dt 200GB (suggested) Locl, high-performnce storge. Appliction dt ckups 150GB (suggested) A remote file server comptile with XFS or locl storge formtted s n XFS filesystem. If the storge is locl, ensure tht it does not cuse contention with the storge for Control Center internl services. The storge for Docker dt nd ppliction dt is configured s LVM thin pools in susequent procedures. 14

Instlling mster host The suggested minimum sizes for ppliction dt nd ppliction dt ckups should e replced with sizes tht meet your ppliction requirements. To clculte the pproprite sizes for these storge res, use the following guidelines: Appliction dt storge includes spce for oth dt nd snpshots. The defult se size for dt is 100GB, nd the recommended spce for snpshots is 100% of the se size. Adding the two yields the suggested minimum size of 200GB. For ppliction dt ckups, the recommended spce is 150% of the se size for dt. The following sections provide procedures for formtting nd mounting locl storge s XFS filesystems for Control Center internl services dt nd ppliction dt ckups. Creting filesystem for Control Center internl services This procedure requires one unused device or prtition. Use this procedure to crete n XFS filesystem on n unused device or prtition. 1 Log in to the trget host s root, or s user with superuser privileges. 2 Identify the trget device or prtition for the filesystem to crete. lslk -p --output=name,size,type,fstype,mountpoint 3 Crete n XFS filesystem. Replce Storge with the pth of the trget device or prtition: mkfs.xfs Storge 4 Enter the following commnd to dd n entry to the /etc/fst file. Replce Storge with the pth of the device or prtition used in the previous step: echo "Storge /opt/serviced/vr/isvcs xfs defults 0 0" >> /etc/fst 5 Crete the mount point for internl services dt. mkdir -p /opt/serviced/vr/isvcs 6 Mount the filesystem, nd then verify it mounted correctly. mount - && mount grep isvcs Exmple result: /dev/xvd1 on /opt/serviced/vr/isvcs type xfs (rw,reltime,ttr2,inode64,noquot) Creting filesystem for ppliction dt ckups This procedure requires one unused device or prtition, or remote file server tht is comptile with XFS. Use this procedure crete n XFS filesystem on device or prtition, or to mount remote filesystem, for ppliction dt ckups. Note If you re using prtition on locl device for ckups, ensure tht the storge for Control Center internl services dt is not on the sme device. 15

Control Center Instlltion Guide 1 Log in to the trget host s root, or s user with superuser privileges. 2 Optionl: Identify the trget device or prtition for the filesystem to crete, if necessry. Skip this step if you re using remote file server. lslk -p --output=name,size,type,fstype,mountpoint 3 Optionl: Crete n XFS filesystem, if necessry. Skip this step if you re using remote file server. Replce Storge with the pth of the trget device or prtition: mkfs.xfs Storge 4 Crete n entry in the /etc/fst file. Replce File-System-Specifiction with one of the following vlues: the pth of the device or prtition used in the previous step the remote server specifiction echo "File-System-Specifiction \ /opt/serviced/vr/ckups xfs defults 0 0" >> /etc/fst 5 Crete the mount point for ckup dt. mkdir -p /opt/serviced/vr/ckups 6 Mount the filesystem, nd then verify it mounted correctly. mount - && mount grep ckups Exmple result: /dev/sd3 on /opt/serviced/vr/ckups type xfs (rw,reltime,seclel,ttr2,inode64,noquot) Prepring the mster host operting system Perform the steps in Downloding nd stging required files on pge 8, efore performing this procedure. Use this procedure to prepre RHEL/CentOS host s Control Center mster host. 1 Log in to the cndidte mster host s root, or s user with superuser privileges. 2 Disle the firewll, if necessry. This step is required for instlltion ut not for deployment. For more informtion, refer to the Control Center Plnning Guide. Determine whether the firewlld service is enled. systemctl sttus firewlld.service If the result includes Active: inctive (ded), the service is disled. Proceed to the next step. If the result includes Active: ctive (running), the service is enled. Perform the following sustep. 16

Instlling mster host Disle the firewlld service. systemctl stop firewlld && systemctl disle firewlld On success, the preceding commnds disply messges similr to the following exmple: rm '/etc/systemd/system/dus-org.fedorproject.firewlld1.service' rm '/etc/systemd/system/sic.trget.wnts/firewlld.service' 3 Optionl: Enle persistent storge for log files, if desired. By defult, RHEL/CentOS systems store log dt only in memory or in ring uffer in the /run/log/ journl directory. By performing this step, log dt persists nd cn e sved indefinitely, if you implement log file rottion prctices. For more informtion, refer to your operting system documenttion. Note The following commnds re sfe when performed during n instlltion, efore Docker or Control Center re instlled or running. To enle persistent log files fter instlltion, stop Control Center, stop Docker, nd then enter the following commnds. mkdir -p /vr/log/journl && systemctl restrt systemd-journld 4 Enle nd strt the Dnsmsq pckge. The pckge fcilittes networking mong Docker continers. systemctl enle dnsmsq && systemctl strt dnsmsq If nme resolution in your environment relies solely on entries in /etc/hosts, configure dsnmsq so tht continers cn use the file: c d e f g h i j k Open /etc/dnsmsq.conf with text editor. Locte the line tht strts with #domin-needed, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Locte the line tht strts with #ogus-priv, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Locte the line tht strts with #locl=/loclnet/, nd then mke copy of the line, immeditely elow the originl. Remove net, nd then remove the numer sign chrcter (#) from the eginning of the line. Locte the line tht strts with #domin=exmple.com, nd then mke copy of the line, immeditely elow the originl. Replce exmple.com with locl, nd then remove the numer sign chrcter (#) from the eginning of the line. Sve the file, nd then close the editor. Restrt the dnsmsq service. systemctl restrt dnsmsq 5 Instll nd configure the NTP pckge. Note This procedure ssumes the host hs internet ccess. To instll nd configure NTP on host tht does not hve internet ccess, see Configuring privte mster NTP server on pge 86. 17

Control Center Instlltion Guide c d Instll the pckge. yum instll ntp Set the system time. ntpd -gq Enle the ntpd demon. systemctl enle ntpd Configure ntpd to strt when the system strts. Currently, n unresolved issue ssocited with NTP prevents ntpd from restrting correctly fter reoot. The following commnds provide workround to ensure tht it does. echo "systemctl strt ntpd" >> /etc/rc.d/rc.locl chmod +x /etc/rc.d/rc.locl Instlling Docker CE nd Control Center Perform the steps in Downloding nd stging required files on pge 8 efore performing this procedure. Use this procedure to instll Docker CE nd Control Center on host. 1 Log in to the host s root, or s user with superuser privileges. 2 Instll Docker CE 17.09.0 from the locl repository mirror. Instll Docker CE. yum instll --enlerepo=zenoss-mirror docker-ce-17.09.0.ce If yum returns n error due to dependency issues, see Resolving pckge dependency conflicts on pge 89 for potentil resolutions. Enle utomtic strtup. systemctl enle docker 3 Instll Control Center 1.5.0 from the locl repository mirror. Instll Control Center. yum instll --enlerepo=zenoss-mirror \ /opt/zenoss-repo-mirror/serviced-1.5.0-1.x86_64.rpm If yum returns n error due to dependency issues, see Resolving pckge dependency conflicts on pge 89 for potentil resolutions. Enle utomtic strtup. systemctl enle serviced 4 Mke ckup copy of the Control Center configurtion file. Mke copy of /etc/defult/serviced. cp /etc/defult/serviced /etc/defult/serviced-1.5.0-orig 18

Instlling mster host Set the ckup file permissions to red-only. chmod 0440 /etc/defult/serviced-1.5.0-orig Configuring Docker Use this procedure to configure Docker. 1 Log in to the mster host s root, or s user with superuser privileges. 2 Crete symolic link for the Docker temporry directory. Docker uses its temporry directory to spool imges. The defult directory is /vr/li/docker/tmp. The following commnd specifies the sme directory tht Control Center uses, /tmp. You cn specify ny directory tht hs minimum of 10GB of unused spce. Crete the docker directory in /vr/li. mkdir /vr/li/docker Crete the link to /tmp. ln -s /tmp /vr/li/docker/tmp 3 Crete systemd drop-in file for Docker. Crete the override directory. mkdir -p /etc/systemd/system/docker.service.d Crete the unit drop-in file. ct <<EOF > /etc/systemd/system/docker.service.d/docker.conf [Service] TimeoutSec=300 EnvironmentFile=-/etc/sysconfig/docker ExecStrt= ExecStrt=/usr/in/dockerd \$OPTIONS TsksMx=infinity EOF c Relod the systemd mnger configurtion. systemctl demon-relod 4 Crete n LVM thin pool for Docker dt. For more informtion out the serviced-storge commnd, see serviced-storge on pge 71. To use n entire lock device or prtition for the thin pool, replce Device-Pth with the device pth: serviced-storge crete-thin-pool docker Device-Pth To use 50GB of n LVM volume group for the thin pool, replce Volume-Group with the nme of n LVM volume group: serviced-storge crete-thin-pool --size=50g docker Volume-Group On success, the result is the device mpper nme of the thin pool, which lwys strts with /dev/mpper. 5 Configure nd strt the Docker service. 19

Control Center Instlltion Guide Crete vrile for the nme of the Docker thin pool. Replce Thin-Pool-Device with the nme of the thin pool device creted in the previous step: c d mypool="thin-pool-device" Crete vriles for dding rguments to the Docker configurtion file. The --exec-opt rgument is workround for Docker issue on RHEL/CentOS 7.x systems. mydriver="--storge-driver devicempper" mylog="--log-level=error" myfix="--exec-opt ntive.cgroupdriver=cgroupfs" mymount="--storge-opt dm.mountopt=discrd" myflg="--storge-opt dm.thinpooldev=$mypool" Add the rguments to the Docker configurtion file. echo 'OPTIONS="'$myLog $mydriver $myfix $mymount $myflg'"' \ >> /etc/sysconfig/docker Strt or restrt Docker. systemctl restrt docker The strtup my tke up to minute, nd my fil. If strtup fils, repet the restrt commnd. 6 Configure nme resolution in continers. Ech time it strts, docker selects n IPv4 sunet for its virtul Ethernet ridge. The selection cn chnge; this step ensures consistency. Identify the IPv4 sunet nd netmsk docker hs selected for its virtul Ethernet ridge. c ip ddr show docker0 grep inet Open /etc/sysconfig/docker in text editor. Add the following flgs to the end of the OPTIONS declrtion. Replce Bridge-Sunet with the IPv4 sunet docker selected for its virtul ridge: --dns=bridge-sunet --ip=bridge-sunet/16 For exmple, if the ridge sunet is 172.17.0.1, dd the following flgs: --dns=172.17.0.1 --ip=172.17.0.1/16 Note Use spce chrcter ( ) to seprte flgs, nd mke sure the doule quote chrcter (") delimits the declrtion of OPTIONS. d e Sve the file, nd then close the editor. Restrt the Docker service. systemctl restrt docker Loding imge files Use this procedure to lod imges into the locl Docker registry on host. 20

Instlling mster host 1 Log in to the host s root, or s user with superuser privileges. 2 Chnge directory to /root. cd /root 3 Lod the imges. for imge in instll-zenoss-*.run do /in/echo -en "\nloding $imge..." yes./$imge done 4 List the imges in the registry. docker imges The result should show one imge for ech rchive file. 5 Optionl: Delete the rchive files, if desired. rm -i./instll-zenoss-*.run Creting the ppliction dt thin pool Use this procedure to crete thin pool for ppliction dt storge. The following Control Center configurtion vriles re used to specify ppliction dt thin pools: SERVICED_FS_TYPE Defult: devicempper The driver to mnge ppliction dt storge on the serviced mster host. Only devicempper is supported in production deployments. The only supported storge lyout for the devicempper driver is n LVM thin pool. To crete thin pool, use the serviced-storge utility. To specify the nme of the thin pool device, use the SERVICED_DM_THINPOOLDEV vrile. SERVICED_DM_THINPOOLDEV Defult: (none) The nme of the thin pool device to use with the devicempper storge driver. Note This procedure does not include specific vlue for the size of the thin pool. For more informtion out sizing this resource, refer to the documenttion for your ppliction. Or, use the suggested minimum vlue, 200GB. You cn dd storge to n LVM thin pool t ny time. Perform these steps: 1 Log in to the mster host s root, or s user with superuser privileges. 2 Crete n LVM thin pool for ppliction dt. For more informtion out the serviced-storge commnd, see serviced-storge on pge 71. To use n entire lock device or prtition for the thin pool, replce Device-Pth with the device pth: serviced-storge crete-thin-pool serviced Device-Pth 21

Control Center Instlltion Guide To use 200GB of n LVM volume group for the thin pool, replce Volume-Group with the nme of n LVM volume group: serviced-storge crete-thin-pool --size=200g serviced Volume-Group On success, the result is the device mpper nme of the thin pool, which lwys strts with /dev/mpper. Record the nme for use in the next step. 3 Edit storge vriles in the Control Center configurtion file. c d e f g Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_FS_TYPE vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Locte the line for the SERVICED_DM_THINPOOLDEV vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Set the vlue to the device mpper nme of the thin pool for ppliction dt. Sve the file, nd then close the editor. Proceed to the next chpter nd configure the host. 22

Configuring nd strting the mster host Configuring nd strting the mster host 3 This chpter includes the procedures for configuring Control Center on the mster host, descries the configurtion options tht pply to the mster host, nd includes steps for strting the mster host for the first time. Before instlling Control Center on delegte hosts, perform the procedures in this chpter. Mny configurtion choices depend on ppliction requirements. Before configuring Control Center on the mster host, review your ppliction documenttion. This chpter includes synopses of the configurtion vriles tht ffect the mster host. For more informtion out vrile, see Control Center configurtion vriles on pge 75. Control Center mintennce scripts on the mster host The scripts in the following list re instlled when Control Center is instlled, nd re strted either dily or weekly y ncron. /etc/cron.hourly/serviced This script invokes logrotte hourly, to mnge the files in /vr/log/serviced. This script is required on the mster host only. /etc/cron.dily/serviced This script invokes logrotte dily, to mnge the /vr/log/serviced.ccess.log file. This script is required on the mster host nd on ll delegte hosts. /etc/cron.weekly/serviced-fstrim This script invokes fstrim weekly, to reclim unused locks in the ppliction dt thin pool. The life spn of solid-stte drive (SSD) degrdes when fstrim is run too frequently. If the lock storge of the ppliction dt thin pool is n SSD, you cn reduce the frequency t which this script is invoked, s long s the thin pool never runs out of free spce. An identicl copy of this script is locted in /opt/ serviced/in. This script is required on the mster host only. /etc/cron.d/cron_zenossdpck This script invokes /opt/serviced/in/serviced-zenossdpck, the dtse mintennce script for Zenoss ppliction, every Sundy t midnight. If the Zenoss ppliction is not instlled or is offline, the commnd fils. You cn chnge the dy of the week nd time of dy when the mintennce script is invoked y editing /etc/cron.d/cron_zenossdpck. This script is required on the mster host only. 23

Control Center Instlltion Guide User ccess control Control Center provides rowser interfce nd commnd-line interfce. To gin ccess to the Control Center rowser interfce, users must hve login ccounts on the Control Center mster host. In ddition, users must e memers of the Control Center rowser interfce ccess group, which y defult is the system group, wheel. To enhnce security, you my chnge the rowser interfce ccess group from wheel to ny other group. To use the Control Center commnd-line interfce (CLI) on Control Center host, user must hve login ccount on the host, nd the ccount must e memer of the serviced group. The serviced group is creted when the Control Center RPM pckge is instlled. Note You cn use two different groups to control ccess to the rowser interfce nd the CLI. You cn enle ccess to oth interfces for the sme users y choosing the serviced group s the rowser interfce ccess group. Pluggle Authentiction Modules (PAM) hs een tested nd is recommended for enling ccess to oth the rowser interfce nd the commnd-line interfce. However, the PAM configurtion must include the sudo service. Control Center relies on the host's sudo configurtion, nd if no configurtion is present, PAM defults to the configurtion for other, which is typiclly too restrictive for Control Center users. For more informtion out configuring PAM, refer to your operting system documenttion. Adding users to the defult rowser interfce ccess group Use this procedure to dd users to the defult rowser interfce ccess group of Control Center, wheel. Note Perform this procedure or the next procedure, ut not oth. 1 Log in to the host s root, or s user with superuser privileges. 2 Add user to the wheel group. Replce User with the nme of login ccount on the mster host. usermod -G wheel User Repet the preceding commnd for ech user to dd. Configuring regulr group s the Control Center rowser interfce ccess group Use this procedure to chnge the defult rowser interfce ccess group of Control Center from wheel to non-system group. The following Control Center vriles re used in this procedure: SERVICED_ADMIN_GROUP Defult: wheel The nme of the Linux group on the serviced mster host whose memers re uthorized to use the serviced rowser interfce. You my replce the defult group with group tht does not hve superuser privileges. SERVICED_ALLOW_ROOT_LOGIN Defult: 1 (true) 24

Configuring nd strting the mster host Determines whether the root user ccount on the serviced mster host my e used to gin ccess to the serviced rowser interfce. Note Perform this procedure or the previous procedure, ut not oth. 1 Log in to the host s root, or s user with superuser privileges. 2 Crete vrile for the group to designte s the dministrtive group. In this exmple, the group is ccuser. You my choose different group, or choose the serviced group. (Choosing the serviced group llows ll rowser interfce users to use the CLI.) mygroup=ccuser 3 Crete new group, if necessry. groupdd $mygroup 4 Add one or more existing users to the group. Replce User with the nme of login ccount on the host: usermod -G $mygroup User Repet the preceding commnd for ech user to dd. 5 Specify the new dministrtive group in the serviced configurtion file. Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_ADMIN_GROUP vrile, nd then mke copy of the line, immeditely elow the originl. c Remove the numer sign chrcter (#) from the eginning of the line. d Chnge the vlue from wheel to the nme of the group you chose erlier. e Sve the file, nd then close the editor. 6 Optionl: Prevent the root user from gining ccess to the Control Center rowser interfce, if desired. Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_ALLOW_ROOT_LOGIN vrile, nd then mke copy of the line, immeditely elow the originl. c Remove the numer sign chrcter (#) from the eginning of the line. d Chnge the vlue from 1 to 0. e Sve the file, nd then close the editor. Enling use of the commnd-line interfce Use this procedure to enle user to perform dministrtive tsks with the Control Center commnd-line interfce. 1 Log in to the host s root, or s user with superuser privileges. 2 Add user to the serviced group. Replce User with the nme of login ccount on the host. usermod -G serviced User Repet the preceding commnd for ech user to dd. 25

Control Center Instlltion Guide Configuring the se size device for tennt dt storge Use this procedure to configure the se size of virtul storge devices for tennts in the ppliction dt thin pool. The se size is used ech time tennt device is creted. In prticulr, the first time serviced strts, it cretes the se size device nd then cretes tennt device from the se size device. The following Control Center configurtion vrile is used to specify the se size device: SERVICED_DM_BASESIZE Defult: 100G The se size of virtul storge devices for tennts in the ppliction dt thin pool, in gigytes. The units symol (G) is required. This vrile is used when serviced strts for the first time, to set the initil size of tennt devices, nd when ckup is restored, to set the size of the restored tennt device. The se size device is sprse device tht occupies t most 1MB of spce in the ppliction dt thin pool; its size hs no immedite prcticl impct. However, the ppliction dt thin pool should hve enough spce for twice the size of ech tennt device it supports, to store oth the dt itself nd snpshots of the dt. Since the ppliction dt thin pool is n LVM logicl volume, its size cn e incresed t ny time. Likewise, the size of tennt device cn e incresed, s long s the ville spce in the thin pool cn support the lrger tennt device plus snpshots. Perform these steps: 1 Log in to the mster host s root, or s user with superuser privileges. 2 Identify the size of the thin pool for ppliction dt. The size is required to set n ccurte vlue for the SERVICED_DM_BASESIZE vrile. lvs --options=lv_nme,lv_size grep serviced-pool 3 Edit storge vriles in the Control Center configurtion file. c d Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_DM_BASESIZE vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Chnge the vlue, if necessry. Replce Fifty-Percent with the vlue tht is less thn or equl to 50% of the size of the thin pool for ppliction dt. Include the symol for gigytes, G: SERVICED_DM_BASESIZE=Fifty-PercentG e Sve the file, nd then close the editor. 4 Verify the settings in the serviced configurtion file. grep -E '^\*[A-Z_]+' /etc/defult/serviced Setting the host role to mster Use this procedure to configure host s the mster host. The following configurtion vrile ssigns the host role: SERVICED_MASTER Defult: 1 (true) 26

Configuring nd strting the mster host Assigns the role of serviced instnce, either mster or delegte. The mster runs the ppliction services scheduler nd other internl services. Delegtes run the ppliction services ssigned to the resource pool to which they elong. Only one serviced instnce cn e the mster; ll other instnces must e delegtes. The defult vlue ssigns the mster role. To ssign the delegte role, set the vlue to 0 (flse). This vrile must e explicitly set on ll Control Center hosts. Perform these steps: 1 Log in to the host s root, or s user with superuser privileges. 2 Edit the Control Center configurtion file. Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_MASTER vrile, nd then mke copy of the line, immeditely elow the originl. c Remove the numer sign chrcter (#) from the eginning of the line. d Sve the file, nd then close the editor. 3 Verify the settings in the serviced configurtion file. grep -E '^\*[A-Z_]+' /etc/defult/serviced Optionl: Chnging the locl Docker registry endpoint Use this procedure to configure the mster host with the endpoint of n lterntive locl Docker registry. Control Center includes locl Docker registry, ut you my use n existing registry in your environment, if desired. For more informtion out configuring locl Docker registry, plese refer to Docker documenttion. Note Chnging the locl Docker registry endpoint is rre. Perform this procedure only if you re sure it is necessry nd the lterntive locl Docker registry is lredy ville in your environment. The following configurtion vrile identifies the locl Docker registry endpoint: SERVICED_DOCKER_REGISTRY Defult: loclhost:5000 The endpoint of the locl Docker registry, which serviced uses to store internl services nd ppliction imges. If the defult vlue is chnged, the host's Docker configurtion file must include the --insecureregistry flg with the sme vlue s this vrile. The sfest replcement for loclhost is the IPv4 ddress of the registry host. Otherwise, the fullyqulified domin nme of the host must e specified. Perform these steps: 1 Log in to the mster host s root, or s user with superuser privileges. 2 Edit the Control Center configurtion file. c d e Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_DOCKER_REGISTRY vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. Replce loclhost:5000 with the endpoint of the locl Docker registry. Use the IP ddress or fully-qulified domin nme of the host nd the port numer. Sve the file, nd then close the editor. 27

Control Center Instlltion Guide 3 Verify the settings in the serviced configurtion file. grep -E '^\*[A-Z_]+' /etc/defult/serviced 4 Add the insecure registry flg to the Docker configurtion file. Open /etc/sysconfig/docker in text editor. Add the following flg to the end of the OPTIONS declrtion. Replce Registry-Endpoint with the sme vlue used for the SERVICED_DOCKER_REGISTRY vrile: --insecure-registry=registry-endpoint Note Use spce chrcter ( ) to seprte flgs, nd mke sure the doule quote chrcter (") delimits the declrtion of OPTIONS. c Sve the file, nd then close the editor. 5 Restrt the Docker service. systemctl restrt docker Optionl: Configuring offline use Use this procedure to configure host to operte without internet ccess. The following configurtion vrile informs serviced tht internet ccess is not ville: SERVICED_OUTBOUND_IP Defult: (none) The IPv4 ddress tht delegtes use to connect to the mster host. When no ddress is specified, serviced ttempts to discover its pulic IP ddress y pinging google.com. This vrile must e set on ll Control Center hosts in either of the following scenrios: Control Center is deployed ehind firewll nd google.com is not rechle. Set the vlue to the IPv4 ddress of the mster host. Control Center is deployed in high-vilility cluster. Set the vlue to the virtul IPv4 ddress of the high-vilility cluster (HA-Virtul-IP). Note Setting the Docker HTTP_PROXY or HTTPS_PROXY environment vriles prevents ccess to the IP ddress defined with this vrile. To enle ccess, unset the Docker vriles, nd then reoot the host. Perform these steps: 1 Log in to the host s root, or s user with superuser privileges. 2 Identify the IPv4 ddress of the host. hostnme -i 3 Edit the Control Center configurtion file. c Open /etc/defult/serviced in text editor. Locte the line for the SERVICED_OUTBOUND_IP vrile, nd then mke copy of the line, immeditely elow the originl. Remove the numer sign chrcter (#) from the eginning of the line. 28

Configuring nd strting the mster host d Chnge the vlue to the IPv4 ddress identified in the previous step. e Sve the file, nd then close the editor. 4 Verify the settings in the serviced configurtion file. grep -E '^\*[A-Z_]+' /etc/defult/serviced Mster host configurtion vriles The tles in this section provide n overview of the serviced configurtion vriles tht pply to the Control Center mster host. Set these vriles s required for your environment or pplictions. Best prctices for configurtion files The Control Center configurtion file, /etc/defult/serviced, contins Bsh environment vriles tht re red y the serviced demon strtup script. The following list descries recommended est prctices for its use nd mintennce: 1 When in dout, mke ckup. Before editing, mking ckup copy of the configurtion file is lwys the sfest choice. 2 Copy vrile, then edit the copy. If you need to revert vrile to its defult vlue, you don't hve to leve the file to look it up. 3 Copy nd edit vrile only if the defult vlue needs to e chnged. It's esier to trouleshoot prolems when only non-defult vriles re copied nd edited. 4 Put the first chrcter of the vrile declrtion in the first column of its line. It's esier to grep for settings when ech one strts line. 5 Add customiztions to the top of the file. Customiztions t the end of the file or scttered throughout the file my e overlooked. 6 In high-vilility deployments, the contents of /etc/defult/serviced on the mster nodes must e identicl. Use utility like sum to compre the files quickly. Storge vriles The vriles in the following tle re set only on the mster host. Use one of the first two groups of vriles ut not oth. Before strting the mster host for the first time, you might need to chnge the defults of the third group. Typiclly, the defults of the lst two groups of vriles re not chnged until Control Center hs mnged n ppliction for while nd need rises. The SERVICED_STORAGE_STATS_UPDATE_INTERVAL vrile sets the intervl for collecting kernel sttistics out the ppliction dt thin pool. Its defult vlue is unlikely to require chnge until need rises. Vrile Description Purpose SERVICED_FS_TYPE SERVICED_DM_ARGS SERVICED_DM_BASESIZE SERVICED_DM_THINPOOLDEV SERVICED_FS_TYPE SERVICED_DM_LOOPDATASIZE SERVICED_DM_LOOPMETADATASIZE The specifictions of devicempper-sed ppliction dt storge resource for production use. The specifictions of devicempper-sed Provide sic informtion out the dt storge resource. Provide sic informtion out the dt storge resource. 29

Control Center Instlltion Guide Vrile Description Purpose SERVICED_ALLOW_LOOP_BACK SERVICED_ISVCS_PATH SERVICED_VOLUMES_PATH SERVICED_BACKUPS_PATH SERVICED_SNAPSHOT_TTL SERVICED_SNAPSHOT_USE_PERCENT SERVICED_MAX_DFS_TIMEOUT SERVICED_LOGSTASH_MAX_DAYS SERVICED_LOGSTASH_MAX_SIZE SERVICED_LOGSTASH_CYCLE_TIME ppliction dt storge resource for development use. The dt storge pths of seprte functionl components of Control Center internl services. The snpshot retention intervl, the percentge of the dt storge thin pool tht is unused, nd the snpshot ttempt timeout intervl. The vriles tht mnge the mount of spce used y the ppliction log storge service. Enle seprte storge res for one or more components. The defult instlltion process puts ll three components on the sme device. Prevent the cretion of snpshots tht re too lrge to fit the thin pool. Prevent ppliction logs from filling the storge device tht logstsh uses. Internl services endpoint vriles The vriles in the following tle must e set identiclly on ll Control Center delegte hosts. The SERVICED_AUTH_TOKEN_EXPIRATION vrile ffects RPC, mux, nd internl services endpoint trffic. Vrile Endpoint Description SERVICED_DOCKER_REGISTRY (vries) The locl Docker registry for Control Center internl services imges nd ppliction imges. SERVICED_ENDPOINT SERVICED_LOG_ADDRESS SERVICED_LOGSTASH_ES SERVICED_STATS_PORT Mster-Host:4979 The serviced RPC server. The endpoint port numer must mtch the vlue of SERVICED_RPC_PORT. Mster-Host:5042 The logstsh service. Mster-Host:9100 The Elsticserch service for logstsh. Mster-Host:8443 The serviced metrics consumer service. SERVICED_AUTH_TOKEN_EXPIRATION (none) The length of time delegte uthentiction token is vlid. RPC service vriles The vriles in the following tle must e set identiclly on ll Control Center hosts, except: SERVICED_RPC_PORT, set only on the mster SERVICED_MAX_RPC_CLIENTS, set only on delegtes 30