A WEB-BASED SOLUTION TO VISUALIZE OPERATIONAL MONITORING LINUX CLUSTER FOR THE PROTODUNE DATA QUALITY MONITORING CLUSTER

Transcription

1 A WEB-BASED SOLUTION TO VISUALIZE OPERATIONAL MONITORING LINUX CLUSTER FOR THE PROTODUNE DATA QUALITY MONITORING CLUSTER BADISA MOSESANE EP-NU Supervisor: Nektarios Benekos Department: EP-NU

2 Table of Contents Abstract 2 1. Introduction 3 2. System Overview 3 3. Key concepts of Puppet and Foreman as used in Neutrino cluster 4 4. Getting Started Creating a CERN CentOS 7 Puppet-managed OpenStack virtual machine Using Facter to display VM facts / information 6 5. Creating Puppet manifests that apply to npcmp hostgroup Triggering Puppet to run as a background daemon Using Hiera to configure modules Installing packages using Puppet Adding Physical nodes in Foreman Visualizing reports from Puppet Conclusion References 13 1

3 Abstract The Neutrino computing cluster made of 300 Dell PowerEdge 1950 U1 nodes serves an integral role to the CERN Neutrino Platform (CENF). It represents an effort to foster fundamental research in the field of Neutrino physics as it provides data processing facility. We cannot begin to over emphasize the need for data quality monitoring coupled with automating system configurations and remote monitoring of the cluster. To achieve these, a software stack has been chosen to implement automatic propagation of configurations across all the nodes in the cluster. The bulk of these discusses and delves more into the automated configuration management system on this cluster to enable the fast online data processing and Data Quality (DQM) process for the Neutrino Platform cluster (npcmp.cern.ch). 2

4 1. Introduction One of the critical tasks during the protodune operation period is the Data Quality Monitoring (DQM) and Prompt processing. To run and maintain the data quality monitoring process, a set of specific hardware of Linux server nodes has to be installed at experimental hall (EHN1). The neutrino cluster consists of 300 U1 nodes each with 16 cores, 16 GB RAM and a CERN CentOS (CC7) Linux has been installed as OS. There is an evident need for an automated configuration management system for this cluster as monitoring of such infrastructure can be cumbersome. To manage the machines hosted in EHN1, some virtual machine (VM) servers are hosted on CERN OpenStack Cloud infrastructure. A Puppet-managed virtual machine was used and configured with Puppet to provide a longer-lived service to users. In the Neutrino cluster, open source Puppet helps to describe machine configurations in a declarative language, bring machines to a desired state and keep them there through automation. The deployment type of Puppet in this cluster was agent/master architecture. Puppet was used with Foreman, an open source tool that helps with the management of servers by an easy interaction with Puppet to automate tasks and application deployment. Foreman provides a robust web interface that allows us to provision, configure cluster nodes and leverage its External Node Classifier (ENC) and reporting capabilities to ease the management of Puppet. For a fully operational cluster these tasks must be carried out; implementation and monitoring of all the necessary web-paged tools for the DQM cluster operational monitoring, including: frontend configuration, server nodes and users monitoring, job monitoring, batch process monitoring, sites availability, data management and transfers between CERN-EOS and EHN1 and the outside world. 2. System Overview The Neutrino cluster configuration management is based on Puppet and its suite of tools. The main components include; facter, puppetdb, puppet manifests, Hiera (data), Foreman and mcollective. A puppet run occurs several times a day in all the nodes. All hosts are registered in Foreman service, a web-front end for the configuration management system which states hostgroups (cluster of machines) where each host belongs to. Nodes periodically use a local puppet agent daemon to ask for puppet master configuration to configure themselves. The puppet master then asks Foreman service which hostgroup the asking node belongs to. Puppet master finds out the configuration of hostgroups (e.g. packages needed for web server) by reading puppet manifests and hiera (data) for that hostgroup stored in a Git repository. The Puppet master compiles desired configuration for a node and hands it back to the nodes Puppet agent to apply it. If the node had any configurations that were out of step, then the local puppet agent applies the appropriate changes else the Puppet agent does nothing. 3

5 Foreman uses the concept of hostgroups to group nodes into a single IT service. The Neutrino hostgroup (npcmp) is divided into sub-hostgroups to allow the service manager to split up the sub components of the service. The sub-hostgroups are: npcmp/development which has development nodes, npcmp/balance for load balancing nodes, npcmp/frontend for web frontend, npcmp/htc for HTCondor batch system and npcmp/workers which are compute nodes. The sub-hostgroups are as shown in the image below. To access the hostgroups one must be a member of np-cmp-admin e-group and can use this link: 3. Key concepts of Puppet and Foreman as used in Neutrino cluster NPCMP Hostgroup: this is a property which is set on each node in Foreman when the node is registered. All nodes in the npcmp hostgroup are all part of the same service, they have common configuration and are managed by the same group of people i.e. np-cmpadmin e-group. Manifests: files written in the Puppet DSL language which Puppet compiles on every run for each node. We have two types of manifests, npcmp manifests located in gitlab repository ( describes nodes in the npcmp hostgroup and module manifests; re-usable units of code, typically each configuring one OS daemon or OS feature. The manifests are stored and versioned using the git versioning system. 4

6 Environments: collections of modules and hostgroups at different development levels. They are defined in YAML files inside Git repository. The Puppet masters use this repository to check out the correct modules and hostgroups for each environment. A machine belongs to one and only one environment (usually production). Nodes in this environment get the released production version of all hostgroup and module code. In the qa environment, nodes get the pre-release version of all hostgroup and module code. Change Control: A critical part managing services for our cluster is Change Control process (QA process), designed to give maximum visibility to configuration changes going through the system. All changes on shared modules are tested by the author in a feature branch then merged into qa environment to inform others about the change. Hiera: A key/value store with a hierarchical search path. The configuration values (data) used by Puppet to describe nodes are put here. These are also stored (and versioned) using the Git versioning system. 4. Getting Started 4.1 Creating a CERN CentOS 7 Puppet-managed OpenStack virtual machine First, we created a second level hostgroup (npcmp) in Foreman where our virtual machines are going to land in. Next SSH into aiadm and clone the git repository containing all the bits of the npcmp hostgroup in the AFS directory. In the cloud panel, download the keypair npcmpkey under the access & security tab, run the command shown below to make the VM known to Foreman, boot it and wait for the first two Puppet reports. We spawned 4 virtual machines np-cmp-vmldb-01, np-cmp-vmldb-02, np-cmpvmldb-03, np-cmp-vmldb-04 and instances are displayed in the OpenStack EP npcmp project. 5

7 4.2 Configuring Git environment Our it-puppet-hostgroup-npcmp in Gitlab heavily uses Git version control system to manage Puppet nodes in the neutrino cluster. This helps coordinate developers work to avoid too many conflicts when merging code to masters. 4.3 Using Facter to display VM facts / information The VMs are ready and configured with Puppet, we now can navigate to Foreman to see the latest Puppet reports of each VM. Once inside our VM, we can use the facter command to display facts about the system such as operating system, the domain, etc. A snippet of output of the above command is as shown below; 6

8 5. Creating Puppet manifests that apply to npcmp hostgroup We created a skeleton puppet manifest inside the code/manifests/npcmp.pp directory and added first lines of puppet code inside the directory where the git repo was cloned. In our subhostgroup that takes care of the batch system, we have made a manifest to the htc sub-hostgroup. After making this manifest we add to the staging area, commit and push to Git. These new codes become visible to Puppet master by less than a minute. 7

9 5.1 Triggering Puppet to run as a background daemon Since puppet can take about an hour to run as a daemon in background it is useful to trigger a Puppet run manually. On our VM, np-cmp-vmlbd-01, we get on a root shell and force a run to apply new changes made in the manifest. First SSH into root@np-cmp-vmlbd-01.cern.ch and run the command puppet agent -t -v. The base configuration done on the manifest file has been fully applied by Puppet. The highlighted notify message shown in the above screenshot was edited in the init.pp file inside itpuppet-hostgroup-npcmp/code/manifests/init.pp 5.2 Using Hiera to configure modules Modules are configurable via Hiera variables that are set at hostgroup level. Puppet has a set of reusable modules maintained by several IT groups that can be configured on our nodes. AFS and SSSD were configured using existing modules so our VMs have access to AFS and SSSD and has an interactive shell. It is worth noting that all the relevant monitoring bits for the new running services are brought in by Puppet automatically. 8

10 SSSD was configured and AFS was turned off in the load balancing client. Other modules can be configured in the manifest in a similar manner. The configured SSSD can be used to grant shell access to a given user. The image below demonstrates how to give a user access to the machines in a hostgroup. 5.3 Installing packages using Puppet Puppet does away with the worry of dependencies as it can install packages on behalf of the user. All Puppet-managed boxes, CERN CentOS and EPEL repositories are all managed by OSrepos module. This module activates yum distro-sync in operating systems of the RedHat family with major versions 6 and above. When using this module, repositories will be installed in /etc/yumpuppet.repos.d/. Examples of packages to be installed using Puppet are Ganglia, Nagios, EOSclient etc. 9

11 We tell Puppet to install this package to the hostgroup manifest. More packages can be installed and availed to all nodes in a similar fashion. 5.4 Adding Physical nodes in Foreman To add a physical machine in Foreman, we set the desired name, hostgroup, interfaces (IP, MAC, FQDN), operating system. Foreman resolves a kickstart template for the physical host triggering an installation by first, preparing the host for installation and rebooting the box to begin installation process. After setting the desired parameters, OS, interfaces and additional information we can use a CLI to perform a similar task by running ai-installhost and ai-remote-power-control command on 10

12 aiadm. Physical machines must be booted over the network and to start the installation we have to reboot the physical machine that was added as a new host in Foreman. 5.5 Visualizing reports from Puppet Foreman allows us to visualize reports from Puppet runs on each individual physical node in our Neutrino cluster. The reports tab shown in the image allows to zoom in to the activity of the nodes, view applied, restarted, failed, skipped and pending events. Foreman provides a way to view the host configuration chart and the status of hosts in the cluster. Among other things it shows hosts that had performed modifications without error, hosts in error state, good host reports in the last 1 day, hosts that had pending changes, out of sync hosts, hosts with no reports and finally hosts with alerts disabled. Run distribution graph shows the number of clients over time in a space of 1500 minutes. 11

13 6. Conclusion The Neutrino computing cluster has a noticeable development progress as now most of the nodes which were bare metals are now running the CERN CentOS 7 (CC7) operating system. The configuration management system of the cluster was done based on Puppet and Foreman for its reporting capabilities. Part of the work or tasks to be done in the future for this cluster is to setup a notification system to alert users and/or admins of the Neutrino system in real time information about the nodes and jobs. The batch system for the Neutrino cluster is HTCondor which is a specialized workload management system for compute-intensive jobs. A cluster monitoring software will be configured to monitor the neutrino cluster as it has the capability to scale as we envision to add more nodes to the cluster. 12

14 7. References CERN. (2017). CERN Configuration Management System User Guide. [online] Available at: [Accessed 9 Aug. 2017]. CERN. (2017). DpmSetupPuppetInstallation. [online] Available at: [Accessed 9 Aug. 2017]. CERN. (2017). CERN ops. [online] Available at: [Accessed 9 Aug. 2017]. 13