Perforce Server Administrator's Guide: Cluster Management

Transcription

1 Perforce Server Administrator's Guide: Cluster Management May 2015

2 Perforce Server Administrator's Guide: Cluster Management May 2015 Copyright 2015 Perforce Software. All rights reserved. Perforce software and documentation is available from You can download and use Perforce programs, but you can not sell or redistribute them. You can download, print, copy, edit, and redistribute the documentation, but you can not sell it, or sell any documentation derived from it. You can not modify or attempt to reverse engineer the programs. This product is subject to U.S. export control laws and regulations including, but not limited to, the U.S. Export Administration Regulations, the International Traffic in Arms Regulation requirements, and all applicable end-use, end-user and destination restrictions. Licensee shall not permit, directly or indirectly, use of any Perforce technology in or by any U.S. embargoed country or otherwise in violation of any U.S. export control laws and regulations. Perforce programs and documents are available from our Web site as is. No warranty or support is provided. Warranties and support, along with higher capacity servers, are sold by Perforce Software. Perforce Software assumes no responsibility or liability for any errors or inaccuracies that might appear in this book. By downloading and using our programs and documents you agree to these terms. Perforce and Inter-File Branching are trademarks of Perforce Software. All other brands or product names are trademarks or registered trademarks of their respective companies or organizations. Any additional software included within Perforce software is listed in License Statements on page 103.

3 Table of Contents Preface... vii About this manual... vii What's new in this guide for viii Major changes... viii Updates and corrections... viii For more information... ix Syntax conventions... ix Please give us feedback... x Chapter 1 Overview... 1 What is server clustering?... 1 Extending the cluster for remote sites... 2 Supporting technologies... 3 Automating failover... 4 Providing ease of use... 5 Hardware and platform requirements... 6 User workflow... 6 Chapter 2 Installation and Set Up... 9 Overview... 9 Preparing for installation Installation requirements Setting up the environment Installing P4CMGR package CentOS and Red Hat Ubuntu Seeding a cluster from an existing server jnltool.pl Prerequisites Create a checkpoint Create a jnltool.pl configuration file Test the configuration file Create a checkpoint file for each cluster node Create and seed the cluster Create and replay journals if needed Redefine configurables and triggers Complete cluster seeding Setting up the server cluster Add depot standby Add workspace servers Add the cluster router Perforce Server Administrator's Guide: Cluster Management iii

4 Perforce Server Administrator's Guide: Cluster Management Create a checkpoint Filesystem layout Upgrading the cluster Specifying workspaces for new users Setting the location and format of log files Creating a client Adding a license file Using a server cluster in a distributed environment Securing the cluster and authenticating users Troubleshooting installation issues Chapter 3 Configuring and Reconfiguring a Server Cluster Cluster-related configuration variables Variable scope Changing configuration values Configurables reference Zookeeper configurables Changing server cluster topology Chapter 4 Backup and Recovery Automatic failover Journal management and the failover process Tuning failover processing Failover and multiple workspaces Backing up workspace servers Backing up an offline workspace server Backing up a workspace server using a replica Recovering failed servers Restoring a failed depot master Adding a new standby after failover Restoring a failed depot standby Restoring a failed workspace server Restoring a failed cluster router Recovering a failed node Temporary failure Permanent failure Troubleshooting failover Scheduled backups Chapter 5 Monitoring and managing server cluster Monitoring a server cluster Getting status information Evaluating replication status Monitoring Zookeeper iv Perforce Server Administrator's Guide: Cluster Management

5 Perforce Server Administrator's Guide: Cluster Management Monitoring events Managing a server cluster Improving performance Managing failovers Chapter 6 Command Reference Using p4cmgr commands Getting authorization to run p4cmgr commands Restrictions on using p4cmgr and p4 commands Getting help for p4cmgr commands General guidelines Specifying server id's Specifying host names Using environment variables Triggers p4cmgr p4cmgr add p4cmgr backup p4cmgr enable p4cmgr init p4cmgr remove p4cmgr restart p4cmgr ssh-reset p4cmgr start p4cmgr status p4cmgr stop p4cmgr upgrade License Statements Index Perforce Server Administrator's Guide: Cluster Management v

6 vi Perforce Server Administrator's Guide: Cluster Management

7 Preface A Perforce cluster provides high performance and automated failover. This book, Perforce Server Administrator's Guide: Cluster Management describes how administrators can install, configure, and maintain a Perforce cluster. This guide assumes familiarity with the material in the Perforce Server Administrator's Guide: Fundamentals. Read this preface for the following information: The contents of this manual The new features introduced in this release The documents you can consult for additional information The syntax conventions used in Perforce documentation How to return feedback About this manual This manual includes the following chapters: Chapter Chapter 1, Overview on page 1 Chapter 2, Installation and Set Up on page 9 Contents Introduces Perforce's server cluster architecture: it describes its advantages, it discusses the technologies used to support it, it explains how you can extend the cluster, and it outlines the workflow used to install and use a cluster. Lists installation pre-requisites, explains how you install a new cluster as well as how you seed a cluster from an existing server. Describes the process of creating a client and setting the location and format of log files. Explains how you use a cluster in a distributed environment, how you migrate from an existing installation, and how you troubleshoot an installation. Chapter 3, Configuring and Reconfiguring a Server Cluster on page 45 Chapter 4, Backup and Recovery on page 55 Describes cluster-related and Zookeeper configurables; explains how you change server cluster topology. Describes how the cluster handles failure, how automatic failover is handled, and what you need to do when components cannot be automatically recovered. Explains how automatic backups are scheduled and how to change the interval when this is done. Perforce Server Administrator's Guide: Cluster Management vii

8 Preface Chapter Chapter 5, Monitoring and managing server cluster on page 67 Chapter 6, Command Reference on page 73 Contents Explains how you monitor the status of the cluster and of some of the underlying technology that supports high availability. Includes information on improving performance and managing failovers. Reference material about the commands you use to set up, monitor, and manage a server cluster. Explains how you get authorization to run p4cmgr commands and provides guidelines for using these commands. What's new in this guide for This section provides a list of changes to this guide for the Perforce Server update release. For a list of all new functionality and major bug fixes in Perforce Server , see the p4cmgr Release Notes. Major changes A cluster can include multiple standby servers A cluster can include multiple routers A cluster can be seeded from an existing standard or commit server The filesystem layout has changed New commands have been added Updates and corrections Adding a new standby after failover Corrected definition for P4JOURNAL The depot master is failed over successively to the available standby servers. The standby server that is most up to date with respect to the depot master is the first selected for failover. Use the p4cmgr add command to add more standby servers. Use the --port option to the p4cmgr add command to specify the port where the additional router listens for requests if you want to assign specific ports. Routers may be placed on the same host or on different hosts. Use the p4cmgr add command to add more routers. See Seeding a cluster from an existing server on page 19 for more information on the use of the jnltool.pl utility in seeding a cluster. See Filesystem layout on page 36 for more information. The p4cmgr enable, p4cmgr ssh-reset, and p4cmgr upgrade commands have been added. For more information, see Chapter 6, Command Reference on page 73. See Adding a new standby after failover on page 61 for more information. See Configurables reference on page 48 for additional information. viii Perforce Server Administrator's Guide: Cluster Management

9 Preface For more information The following table lists and describes key documents for Perforce users, developers, and administrators. For complete information about Perforce documentation, see: For specific information about The basics of Perforce. Installing and administering the Perforce service, including user management, and security settings. Installing and configuring Perforce services (proxies, replicas, and edge servers) in a distributed environment. Installing and administering a Perforce cluster for high performance and automated failover. p4 command line options (reference). P4V, the cross-platform Perforce Visual Client. Perforce plug-ins and integrations. See this documentation Introducing Perforce Perforce Server Administrator's Guide: Fundamentals Perforce Server Administrator's Guide: Multi-site Deployment Perforce Server Administrator's Guide: Cluster Management P4 Command Reference, p4 help P4V User's Guide, P4V online help IDEs: Using IDE Plug-ins Defect trackers: Defect Tracking Gateway Guide Others: online help from the Perforce menu or web site Developing custom Perforce applications using the Perforce C/C++ API. Working with Perforce in Ruby, Perl, Python, and PHP. C/C++ API User's Guide APIs for Scripting Syntax conventions Perforce documentation uses the following syntax conventions to describe command line syntax. Notation literal Meaning Monospace indicates a word or other notation that must be used in the command exactly as shown. Perforce Server Administrator's Guide: Cluster Management ix

10 Preface Notation italics [-f] Meaning Italics indicate a parameter for which you must supply specific information. For example, for a serverid parameter, you must supply the id of the server. Square brackets indicate that the enclosed elements are optional. Omit the brackets when you compose the command. Elements that are not bracketed are required. element1 element2 Ellipses ( ) indicate that the preceding element can be repeated as often as needed. A vertical bar ( ) indicates that either element1 or element2 is required. Please give us feedback We are interested in receiving opinions on this manual from our users. In particular, we'd like to hear from users who have never used Perforce before. Does this guide teach the topic well? Please let us know what you think; we can be reached at <manual@perforce.com>. x Perforce Server Administrator's Guide: Cluster Management

11 Chapter 1 Overview This chapter introduces the Helix server cluster architecture: it describes its advantages, it discusses the technologies used to support it, it explains how you can extend the cluster, and it outlines the workflow used to install and use a cluster. A Helix server cluster is referred to as a Perforce cluster throughout this document. What is server clustering? Server clustering offers a multi-node solution for high availability and horizontal data center scaling. The following figure illustrates the topology of a Perforce server cluster. Figure 1.1. Server Cluster The elements of a Perforce server cluster include the following: A depot master that records changes to revision data. A depot standby that assumes the role of the depot master if the depot master fails. A cluster may include one or more standbys. The depot standby has a copy of the depot master's database and journal files. It does not service workspace server requests. Perforce Server Administrator's Guide: Cluster Management 1

12 Chapter 1. Overview The depot standby is automatically synchronized to the depot master. Should the depot master fail, the depot standby is promoted to master status. Any updates that the depot standby has not recorded from the depot master are rolled back. For example, a submit operation that has completed on the depot master but that the depot standby doesn't know about returns with a failure, and the user needs to re-submit. A fault-tolerant shared filesystem that is accessible to all cluster nodes; it stores archive files and workspace shelves. An optional workspace server that manages client data. You can add more workspace servers to distribute the load of clients across machines. The use of workspace servers is optional. If you are not concerned with offloading some of the work of the depot master or with horizontal scaling, you do not need to create workspace servers. A pre-configured cluster router (Perforce Broker) that dispatches user requests to the workspace servers or directly to the depot master if no workspace servers are used. The broker can also dispatch requests from external edge servers to the depot master. The server cluster is accessed as a single server resource with the P4PORT of the cluster router as the entry point. The router directs incoming requests to the appropriate server; depending on how the cluster is defined, this might be a workspace server or a depot master. A cluster can use more than one router if needed for load balancing or to convey requests from external servers. The client connects to the cluster using the P4PORT on the cluster router, sending requests, and syncing with the depot as if working with a standard Perforce server. Depot masters, standby servers, and workspace servers are clustering implementations of p4d. The router is a clustering implementation of p4broker. Server cluster elements are supported by additional technologies: Apache Zookeeper for automated failover, and SaltStack for administrative ease of use. These are described in Supporting technologies on page 3. Note in Figure 1.1, Server Cluster on page 1, that each member of the cluster also runs a p4zk process that communicates with the Zookeeper service. Its function is to inform Zookeeper of the availability of that cluster member. If you are familiar with the commit-edge model, it might help to think of a server cluster as using a highly available, horizontally scalable commit-edge architecture. The depot master functions as a commit server; the depot standby, as a forwarding replica, and the workspace server, as an edge server. What distinguishes a cluster from the commit-edge architecture is the use of a shared filesystem, the low latency implied in the data center environment, the pre-configuration of the cluster elements that allows them to work together, and the supporting technologies that provide automated failover and ease of use. Extending the cluster for remote sites A server cluster is meant to run in a data center environment; it depends upon a shared filesystem and the low latency that characterizes such an environment. You can, if you need to, extend your cluster geographically, by adding a proxy or an edge server as shown in the next figure: 2 Perforce Server Administrator's Guide: Cluster Management

13 Chapter 1. Overview Figure 1.2. Extending the Cluster in a Distributed Environment For more information on how to configure this deployment, see Using a server cluster in a distributed environment on page 40. Supporting technologies Two features of a cluster, failover and ease of use, are furnished by supporting technologies: Apache Zookeeper and SaltStack. Zookeeper tracks the state of cluster nodes and supports the automatic failover of the depot master to the depot standby. SaltStack provides the distributed configuration management and orchestration platform on which P4CMGR operates. P4CMGR, in turn, is the tool that allows administrators to easily initialize, monitor, and manage the cluster environment. The following sections explain how these technologies support failover and ease of use. Perforce Server Administrator's Guide: Cluster Management 3

14 Chapter 1. Overview Automating failover A server cluster uses an Apache Zookeeper service to automate the process of failure detection and of role migration from the failing depot master to the depot standby. The fault-tolerant Zookeeper service maintains cluster state information and monitors the health of the cluster nodes. The cluster uses the p4zk process, which supports each server in the cluster, to register the existence of the server and to respond to cluster state changes. Three instances of the Zookeeper service must be deployed on three different nodes in the cluster environment to support the fault tolerance of Zookeeper itself. The next figure illustrates the Zookeeper components that are part of your cluster installation. These include the p4zk processes, deployed on all depot nodes, and the three Zookeeper servers: one is found on a dedicated host (upper left corner); the two others are shown by the Z icon on the workspace server node and on the cluster router node. Figure 1.3. Cluster with Supporting Technology At startup, each cluster member launches a new p4zk process that serves as a conduit from that member to Zookeeper. In the event of the failure of a node in the cluster, p4zk is responsible for the following: If a depot master fails, the p4zk process instructs the depot standby to restart as a depot master, and it updates the router's configuration file and the configuration file of each workspace server with the new target that of the promoted depot standby. 4 Perforce Server Administrator's Guide: Cluster Management

15 Chapter 1. Overview It also ensures that no data is lost in the transition to the new depot master. If a workspace server fails, p4zk updates the router's configuration file so that it no longer forwards client requests to that server. If a workspace node is stopped or started, p4zk reports the change to the depot master and it updates the router's configuration file. (In the case of workspace addition or removal, it is P4CMGR that handles the change.) For more detailed information about the failover process, see Chapter 4, Backup and Recovery on page 55. Providing ease of use Setting up and administering a cluster includes many complex tasks that are vulnerable to error: P4CMGR automates administration tasks that relate to cluster setup and management, and it supports the cluster's high availability and horizontal scaling features. P4CMGR is a collection of Python scripts that extends Salt to manage administrative activities in a cluster. Salt support is accessed using the p4cmgr interface, which defines the commands listed in the following table. For detailed information, see the description of each command in Chapter 6, Command Reference on page 73. p4cmgr add p4cmgr backup p4cmgr enable p4cmgr init p4cmgr remove p4cmgr restart Adds a node to a cluster. Checkpoints the depot master using the replicated data on the depot standby. Restarts the cluster and enables the handling of client requests after the cluster is upgraded. Initializes a cluster. Removes a service from a cluster. Stops and then starts one or more servers in a cluster. p4cmgr ssh-reset Resets SSH certificates. p4cmgr start p4cmgr status p4cmgr stop p4cmgr upgrade Starts one or more nodes in a cluster. Returns a debug dump for all hosts and services in a cluster. Stops the specified node(s) in a cluster. Upgrades the cluster. The administrator uses these commands to initialize and manage the server cluster. Note that some p4cmgr functionality takes precedence over equivalent p4 commands. For this reason, the router actively filters certain p4 commands to avoid conflicts between the effect of these Perforce Server Administrator's Guide: Cluster Management 5

16 Chapter 1. Overview commands and the proper functioning of a cluster. Filtered commands include some variants of the p4 admin and p4 configure commands, the p4 cluster command, and the p4 cachepurge command. For detailed information about these commands see, Restrictions on using p4cmgr and p4 commands on page 73. Hardware and platform requirements The cluster environment must include the following: A tightly coupled data center environment with a low latency and stable network. A sufficient number of 64-bit servers or virtual machines to support high availability and if needed, horizontal scaling. Operating systems: Red Hat Linux 6.5, Ubuntu 12.04, or CentOS 6.4. The operating systems can run on a physical or a virtual machine. High availability requires that the depot master and the depot standby be hosted on separate machines. A fault tolerant, POSIX-compliant shared filesystem; no specific requirements for locking semantics. Firewall protection. The cluster needs to be situated behind a firewall; only the administrator may access servers other than the router directly. Inside the firewall, the administrator can issue p4cmgr commands as well as standard p4 commands. For additional details, see Chapter 2, Installation and Set Up on page 9. User workflow The general workflow for installing, initializing, and working with a cluster includes the following steps, which must be performed by a system administrator: 1. Prepare the cluster environment. 2. Install the perforce-cluster-manager package. You can now execute p4cmgr commands. 3. Use the p4cmgr init command to initialize the cluster. This sets up a depot master on the target machine and sets cluster-related configurables on the depot master. It also creates a Zookeeper service on the node where the p4cmgr init is executed. 4. Use the p4cmgr add zookeeper command to add the nodes where Zookeeper is going to run. Before you add the depot standby(s), workspace, or router nodes, you should add the nodes that will host Zookeeper services. You'll need a minimum of three or a maximum of five nodes. Either way, the number must be odd. You should provision the Zookeeper nodes first because every time you add a Zookeeper node, you must stop and restart the cluster. It is easiest to just do this once. You may not have more than one Zookeeper service on one host. If you have an existing Zookeeper service on a host, you need to move it somewhere else. 6 Perforce Server Administrator's Guide: Cluster Management

17 Chapter 1. Overview 5. Use the p4cmgr restart command to restart the cluster. 6. Use the p4cmgr add depot command to add one or more depot standbys. 7. Use the p4cmgr add workspace command for each workspace server you want to create. (Optional) 8. Use the p4cmgr add router command to add a cluster router. 9. Create users, unless you are planning to have the users be auto-created. Service users are automatically created for each server to allow trust between cluster members. Clients can access the server cluster using any Perforce client, including the p4 command line interface, P4V, P4VS, or Eclipse. The next chapter, Chapter 2, Installation and Set Up on page 9, provides detailed information on how you install a server cluster. Perforce Server Administrator's Guide: Cluster Management 7

18 8 Perforce Server Administrator's Guide: Cluster Management

19 Chapter 2 Installation and Set Up Overview Server cluster set up and administration are intended to be fairly straightforward. However, ease of use depends on the integrity of your initial setup, which requires close cooperation between IT staff and the Perforce administrator. Please follow the guidelines and procedures in this chapter closely to ensure the best outcome. A full server cluster consists of the following nodes: One P4CMGR node that also hosts a Zookeeper service. Two additional nodes that host a Zookeeper service each. These are the only services that should be installed on hosts along with other p4d servers. One depot master. One or more depot standbys. One or more workspace servers (optional). One or more cluster routers. A shared archive. Zookeeper services may be hosted on nodes that host other server cluster elements, or they may be hosted on separate nodes. The following figure illustrates a sample deployment: Perforce Server Administrator's Guide: Cluster Management 9

20 Chapter 2. Installation and Set Up Figure 2.1. Deployed Cluster As shown, this cluster contains one P4CMGR node, a depot master, a depot standby (master and standby are both servers of type depot), two workspace servers, a shared archive, and a router. A firewall is recommended in production. Optimal performance is obtained by deploying cluster elements on different networks. The next figure illustrates the ideal deployment with respect to network topology. The four-network configuration is not required; but it is recommended for best performance. 10 Perforce Server Administrator's Guide: Cluster Management

21 Chapter 2. Installation and Set Up Figure 2.2. Network Topology The Perforce administrator is responsible for setting up the cluster and changing the default configuration if needed. However, before this can be done, the IT department must set up the data center environment to support a server cluster; this work is described in the next section. Overall, setting up the cluster from scratch involves the following steps: the first two steps can be performed by the IT department; the last six steps can be done either by the IT or by a Perforce administrator. The Perforce administrator must have root privileges on all cluster nodes. 1. Prepare for installation. 2. Install the P4CMGR package. 3. Initialize the cluster. This creates the depot master, and starts a Zookeeper service on the P4CMGR host. 4. Add Zookeeper on two or four additional nodes. 5. Restart the cluster. 6. Add one or more depot standbys. 7. Add the workspace servers (optional). 8. Add the cluster router. The following sections describe each of these steps in detail. Preparing for installation The following sections describe the work you need to do before you install a cluster. After this preparatory work is completed, the following information should always be available to the administrator responsible for installing and monitoring the cluster: The name of the shared filesystem to be used for archives and shelves. Perforce Server Administrator's Guide: Cluster Management 11

22 Chapter 2. Installation and Set Up The name of the hosts where cluster servers are to be installed and the type of server to be installed there. Important You may not use an IP address in lieu of a host name. Installation requirements The server cluster environment must meet the following requirements: It must be a tightly-coupled data center environment with a network that is stable with low latency. It must include a sufficient number of 64-bit servers or virtual machines to support high availability and if needed, horizontal scaling. If you use virtual machines, observe the following guidelines: The depot master node and its depot standby node must not share the same physical resources other than the shared filesystem. This is to assure high availability in the case of underlying hardware failure. A VM farm should be configured so that VM processes are not restarted on different hosts if the underlying hardware fails. Some VM installations offer high availability features. In this case, the cluster high availability process might conflict with the VM high availability process. To ensure the smooth operation of the cluster, VM migration for high availability should be disabled. Supported Linux operating systems must be installed: Ubuntu 12.04, CentOS 6 (6.4+), or RHEL 6 (6.5+). All machines in the cluster must run the same operating system; but all operating systems do not have to be the same version. The operating systems can run on physical hardware or in a virtual machine. High availability requires that the depot master and the depot standbys be hosted on separate machines. The operating system for the fault tolerant, shared filesystem can be different from the operating system of other nodes in the cluster. Each depot and workspace server needs /p4mnt/shared as the fault tolerant shared file system: archive data and shelves are stored there. You must create this directory on the shared filesystem for the install process to work. In addition, for improved performance, we recommend /p4mnt/root, /p4mnt/journals, and /p4mnt/ logs be created and placed on different disks, as shown in the next table. The directories need to be owned by perforce with permissions 755. The P4CMGR host and router hosts do not need any special disk setup. See Filesystem layout on page 36 for additional information about placing and managing files in a cluster. Filesystem Contents Scope Disk type /p4mnt/shared archive data, shelves, specs, checkpoints cluster shared network disk 12 Perforce Server Administrator's Guide: Cluster Management

23 Chapter 2. Installation and Set Up Filesystem Contents Scope Disk type /p4mnt/root database files server fast read and write: SSD or Raid-10, or local disk /p4mnt/journals live journals, rotate journals, checkpoints server Fast write /p4mnt/logs server logs, zookeeper logs server Fast write The cluster must be protected by a firewall. Only the administrator may access servers in the cluster other than the router directly. Normal users can not use p4cmgr commands, and they must observe some restrictions on the use of p4 commands. For a discussion of these restrictions, see Restrictions on using p4cmgr and p4 commands on page 73. Setting up the environment To set up a server cluster environment, the IT department, in consultation with a Perforce administrator, must do the following: 1. Plan the topology of the deployment, identifying the physical and virtual machines that should host cluster components (depot master, depot standbys, workspace servers, and routers.) 2. Install supported operating systems on the selected physical or virtual machines. 3. Make sure there are no Zookeeper services already running on any of the nodes intended for the server cluster. If there are existing Zookeeper services supporting other technology, these must be moved elsewhere. 4. Make sure there are no SaltStack services already running on any of the nodes intended for the cluster. Move existing SaltStack services elsewhere. 5. Set up authentication for the administrator: Create the operating system user perforce and the operating system group perforce with the same user id and group id on every system in the cluster. Sharing user and group id makes ownership of files on the shared filesystem much easier. Select a group id that is not currently in use on any of your systems. You can see the group id's that are currently used in /etc/group. Select a user id that is not currently in use on any of your systems. You can see the user id's that are currently used in /etc/passwd. We recommend that you set the Perforce operating system user's default shell to be /bin/bash. groupadd -g groupid perforce useradd -s /bin/bash -u userid -g groupid -m perforce Perforce Server Administrator's Guide: Cluster Management 13

24 Chapter 2. Installation and Set Up Make sure that every user who would run p4cmgr commands is a member of the operating system group perforce on the P4CMGR host. An operating system user other than perforce may run p4cmgr commands, but they must be members of the operating system group perforce. Any operating system user who wants to view logs on clustered nodes must also be a member of the operating system group perforce. Whether you plan to run as root or non-root, you should do the following on all your cluster nodes: Use the usermod command to make sure that your operating system user is part of the perforce group. For example: usermod -a -G perforce root usermod -a -G perforce OS_user_with_ssh_and_sudo_privileges Logout and login again after adding your user to the perforce group, or the current shell might not have the group associated with your user. The operating system user who sets up the server cluster or add systems to it by running the p4cmgr init or p4cmgr add commands, must have sudo (NOPASSWD) privileges on all systems in the cluster. This sudo (NOPASSWD) access can be removed after all setup commands have been run. For example: echo 'perforce ALL=(ALL) NOPASSWD:ALL' > /tmp/perforce sudo chmod 0440 /tmp/perforce sudo chown root:root /tmp/perforce sudo mv /tmp/perforce /etc/sudoers.d Warning Failure to have sudo (NOPASSWD) access on all systems when setting up the system results in the p4cmgr init or p4cmgr add command becoming unresponsive. If this happens, you need to stop the stalled commands using Ctrl+C, configure the sudo (NOPASSWD) permission where it is missing and then re-run the p4cmgr init or p4cmgr add command. 6. Make sure P4PASSWD is not set. For example: unset P4PASSWD 7. Configure the network to support the server cluster: All hosts must be able to see (ping) one another on the network. In /etc/hosts ensure that localhost resolves correctly. Remove any entry for in /etc/hosts. 14 Perforce Server Administrator's Guide: Cluster Management

25 Chapter 2. Installation and Set Up If you must have an entry for in /etc/hosts, put it above the entries for the cluster hosts. Make sure that the host name comes before any reference to localhost. Make sure that the fully-qualified domain name is included before any localhost domain entry. This is to ensure that when any Perforce server on this host starts up and binds to the machine's host name, it does not bind to localhost/ as this only allows connections from localhost localhost localhost.localdomain host1.example.com host host2.example.com host2 The fully-qualified host names of any depot master or depot standby are required by Perforce licensing if a license is required for your server cluster. If you use /etc/hosts to manage IP addresses, define all the cluster hosts in /etc/hosts on every cluster host. If you use DNS, make sure all the cluster hosts are registered with DNS. 8. Disable installed firewalls on all server cluster hosts. If this is not possible or desirable, make sure that the following ports are open. Incoming ports: SSH: 22 SaltStack: 4505, 4506 Perforce processes: These ports are for depot master, depot standby, workspace, and router hosts. Not all ports are used; once the cluster is set up, you can reduce this to active/configured ports only. NFS: 111, 2049 Zookeeper: 2380, Zookeeper hosts only, including the P4CMGR host. Outgoing ports (access from hosts to outside the cluster firewall): SMTP: 25 For the P4CMGR host only, to allow crontab to send s as part of the regularly scheduled backup. HTTP: 80 (needed for apt/yum) HTTPS: 443 (needed for apt/yum) FTP: 21 (needed for apt/yum) Perforce Server Administrator's Guide: Cluster Management 15

26 Chapter 2. Installation and Set Up 9. Mount a shared, writable, fault-tolerant filesystem on /p4mnt/shared on the nodes that serve as depot master, depot standby, and workspace servers. This directory must be owned by perforce:perforce and have 770 permissions. 10. If you are using NFS to provide your fault tolerant, shared filesystem, set NFS client configuration options using the values described in the following table. For example, enter the following line in / etc/fstab: NFS-host:shared-dir /p4mnt/shared nfs rw,hard,_netdev,sync,auto,nodev,noexec,nosuid,intr 0 0 rw hard _netdev sync auto nodev noexec nosuid intr Read-write (default). NFS share must be available. This means that NFS requests are retried indefinitely. The filesystem resides on a device that requires network access; this prevents the system from attempting to mount filesystems until the network has been enabled on the system. To protect data integrity, do not use NFS caching on writes. Automatically mount at boot time (default). Perforce does not actually create device nodes for devices. Executables should not be run directly from a Perforce archive directory. Triggers in the archive are copied to a temporary directory before they are executed. SetUID executables should not be run directly from a Perforce archive directory. Allow keyboard interrupts when the server is down. The NFS server should export the archives share as read-write with sync enabled, and no_subtree_check. For example, for NFS v3, you would use a directive like the following: /archives *(rw,sync,no_subtree_check,no_root_squash) For NFS v4, you can use the same directive or you can omit no_root_squash. If you omit it, you need to change the root mapping in /etc/idmapd.conf. In the following example, you must replace perforce with the name of your P4CMGR perforce user.... Nobody-User = perforce Nobody-Group = perforce Perforce Server Administrator's Guide: Cluster Management

27 Chapter 2. Installation and Set Up The no_root_squash option allows P4CMGR to change the ownership of the directories it creates in the shared archive appropriately. If you do not use the no_root_squash option, in NFS v3, installation fails. If you do not use the no_root_squash option in NFS v4, you must remap the nobody user and group to the perforce user and group in /etc/idmapd.conf to allow P4CMGR to change ownership. 11. Optional: Enable SSH trust from the P4CMGR host to all other cluster hosts. This should be done for the user would run the p4cmgr init and p4cmgr add commands. If this is not done in advance, you are prompted to accept an SSH key fingerprint during the execution of the p4cmgr init and p4cmgr add commands. The user for whom trust is established does not need to be root; however, that user does need to have sudo (NOPASSWD) permission on all nodes. Use a command like the following to provide that permission: ssh-keyscan myhost > ~/.ssh/known_hosts 12. If you are installing on Red Hat, you must have the optional RPM repository enabled on all systems in the cluster. (This includes P4CMGR host, Zookeeper, depot master, depot standby, workspace, and router hosts if these are running Red Hat.) Use the following command: sudo subscription-manager repos --enable=rhel-6-server-optional-rpms 13. If you are seeding the cluster from an existing server (checkpoint), you need Perl version 5.14 or later. 14. Enclose the clustered servers within a firewall to prevent non-administrative direct access to the internal servers of the cluster. Standard Perforce users should only be able to access the cluster through the cluster router. (The firewall should be configured to allow access to the P4PORT of the router.) Installing P4CMGR package To install P4CMGR, follow directions given for your operating system, in the next sub-sections. p4cmgr administrative work can be done by logging in as root, or by logging in as a non-root user who is allowed to get a root shell (like sudo bash). The admin must be a member of the perforce group. CentOS and Red Hat Working on the host where you plan to install p4cmgr, follow these steps to install P4CMGR: 1. Make sure the file /etc/yum.repos.d/epel.repo includes the following content. This provides the dependencies required by SaltStack and ensures that Salt packages are installed from Perforce's repository. Perforce Server Administrator's Guide: Cluster Management 17

28 Chapter 2. Installation and Set Up [epel] name=epel baseurl= failovermethod=priority enabled=1 gpgcheck=1 exclude=salt salt-* 2. Make sure the file /etc/yum.repos.d/perforce.repo includes the following content. This ensures that the packages for Perforce packages for the Perforce server, broker, and p4zk as well as our re-hosted Zookeeper and SaltStack packages can be installed. [perforce] name=perforce baseurl= enabled=1 gpgcheck=1 3. Run the following commands to install P4CMGR: sudo rpm --import sudo rpm --import sudo yum update # recommended, not required sudo yum install perforce-cluster-manager-r15.1 You are now ready to set up your cluster. If you want to start from scratch, follow the directions given in Setting up the server cluster on page 30. If you want to seed the cluster from an existing checkpoint, follow the directions given in Seeding a cluster from an existing server on page 19. Ubuntu Working on the host where you plan to install P4CMGR, follow these steps: 1. Make sure the file /etc/apt/sources.list.d/p4cmgr.list contains the following content. deb precise release 2. Make sure the file /etc/apt/preferences.d/salt-perforce-pin-1000 includes the following content. This prefers Perforce Salt-related packages. 18 Perforce Server Administrator's Guide: Cluster Management

29 Chapter 2. Installation and Set Up Package: *salt-* Pin: release o=package.perforce.com Pin-Priority: Run the following commands to install P4CMGR. The first line ensures that no Perforce packages are installed: "rc" packages are okay; "ii" packages are not okay. The sudo with apt-key is not needed if you have root privileges. dpkg -l egrep 'perforce' wget -q -O - sudo apt-key add - sudo apt-get update sudo apt-get install perforce-cluster-manager.r15.1 You are now ready to set up your cluster. If you want to start from scratch, follow the directions given in Setting up the server cluster on page 30. If you want to seed the cluster from an existing checkpoint, follow the directions given in Seeding a cluster from an existing server on page 19. Seeding a cluster from an existing server Having completed the preparatory steps given in the previous sections, you can seed the cluster from an existing server or a group of federated services. This section explains how you do this using the jnltool.pl utility. You use the jnltool.pl utility to seed a cluster based on an existing checkpoint. jnltool.pl takes the checkpoint of a standard or commit server as its input, consults a configuration file defining the servers that make up a cluster, and outputs a set of checkpoints for seeding a master depot, a standby depot, and a workspace server if any. This process is illustrated in the following figure. The seeding checkpoint is specified using the default name checkpoint; the configuration file is specified using the default name checkpoint.yaml. Using this checkpoint and information from the configuration file, jnltool.pl outputs four checkpoints (one for each server in the cluster); the checkpoint files are named by default using the serverid for each server. Because default filenames are used, these do not have to be specified in the jnltool.pl command lines that are shown beneath the figure. Perforce Server Administrator's Guide: Cluster Management 19

30 Chapter 2. Installation and Set Up Figure 2.3. Using jnltool.pl to create checkpoints for cluster The steps required to seed and initialize a cluster are summarized below and described in the subsections that follow. 1. Make sure that you satisfy the prerequisites specified in Prerequisites on page Generate a checkpoint file for the server configuration that you want to use to seed the cluster. See Create a checkpoint on page Use jnltool.pl to create a template for the configuration file that is used to seed the cluster. Edit this file to provide site-specific information. See Create a jnltool.pl configuration file on page Use the --cluster-preview option to the jnltool.pl command to review the client assignments for the cluster. See Create a jnltool.pl configuration file on page Edit the configuration file until all workspaces are mapped to the correct servers. 6. Repeat steps 4 and 5 until you obtain satisfactory results. 7. Run jnltool.pl for each server in the cluster to generate the filtered checkpoint for that server. See Create a checkpoint file for each cluster node on page Place the compressed generated checkpoints in the appropriate directories for use by P4CMGR. (You can create compressed checkpoints using the --gzip-output option to the jnltool.pl command. See Create and seed the cluster on page Perforce Server Administrator's Guide: Cluster Management

31 Chapter 2. Installation and Set Up 9. Follow the directions given in Setting up the server cluster on page 30 to seed and start the cluster. Important Do not checkpoint the depot master when you are done. 10. Run jnltool.pl to generate filtered journals for any server activity that has taken place after you created the original checkpoint, but before you have seeded the cluster. 11. Replay the journals into the corresponding servers. See Create and replay journals if needed on page Examine the jnltool.pl output files to determine whether you need to redefine any configurables or triggers for any of the servers. Adjust configurable and trigger definitions if needed. 13. Checkpoint the depot master. The jnltool.pl, which plays a key role in seeding the cluster, is described next. jnltool.pl You can use jnltool.pl to do a variety of things. In the context of seeding a Perforce cluster, jnltool.pl generates a jnltool.pl configuration file template, provides a preview of checkpoint outputs, generates the checkpoints that you use to seed the cluster, and generates the journal files needed to update the servers after you have seeded the cluster. jnltool.pl is available here: ftp://ftp.perforce.com/perforce/r15.1/bin.tools/ You can install it as follows: On Ubuntu: sudo apt-get install perforce-jnltool On CentOS/Red Hat: sudo yum install perforce-jnltool The jnltool.pl tool syntax includes the following use cases: jnltool.pl --to-cluster --gen-config original_ckp [> myconfig.yaml] jnltool.pl --to-cluster --cfg-file=myconfig.yaml --server-id=myserverid [--gzip-output] [--output-dir=mydir] [--cluster-preview] original_ckp jnltool.pl --to-cluster --cfg-file=myconfig.yaml --server-id=myserverid { --output-dir=mydir --from-ckp=original_ckp } journal.n Perforce Server Administrator's Guide: Cluster Management 21

32 Chapter 2. Installation and Set Up Use the first syntax option to generate a configuration file that contains the information jnltool.pl needs to create the filtered checkpoints and the journals to seed your cluster. Use the second syntax option to preview or to generate a checkpoint for each server in your cluster. Use the third option to generate journals that you can replay to bring the cluster up to date with the server you're seeding from. Options to this utility command are described in the following table: --to-cluster Filter the original checkpoint to generate multiple checkpoints and journals to seed a cluster. --cfg-file=myconfig.yaml The name of the YAML configuration file that holds information defining the cluster and the names of jnltool.pl output files. See Create a jnltool.pl configuration file on page 24 for more information. Default value: original_checkpoint_name.yaml --from-ckp=original_ckp Converting a checkpoint and creating the initial cluster databases might take a while. If the source server has remained active, the journals created since the imported checkpoint also need to be converted and imported into the cluster nodes to bring those up to date. The third syntax option above does this. The specified checkpoint is used by jnltool.pl to find cached information that it uses to filter the input journal file specified with the journal_file.n parameter. See Create and replay journals if needed on page 29 for more information. --gen-config --server-id=myserverid --gzip-output Write configuration information to STDOUT based on the data in the supplied checkpoint. You can redirect output to a file if you wish. See Create a jnltool.pl configuration file on page 24 for more information. The id of the server for which the checkpoint is being generated. You need to run jnltool.pl once for each server to generate the appropriate checkpoint for that server. Gzip the output checkpoint files. By default, the checkpoint files are uncompressed. This process is faster if the pigz or gzip programs are installed ( zlib.net/pigz/). --output-dir=mydir Specify the directory where the output files should be placed. This location can be overridden by files with absolute paths in the config output section. 22 Perforce Server Administrator's Guide: Cluster Management

33 Chapter 2. Installation and Set Up By default output files are placed in the same directory as that of the input checkpoint, with the checkpoint name as a prefix. For example, checkpoint.1 would have checkpoint.1-db.config for the db.config table dump. --cluster-preview original_ckp myjournal.n Do not create the output checkpoint. Use this option to preview command output. See Test the configuration file on page 28 for more information. The name of the input checkpoint file. This file should not be compressed. The name of the journal file that you want filtered for the server specified by myserverid. See Create and replay journals if needed on page 29 for more information. The following example illustrates the creation of a checkpoint for the server ws1 from the checkpoint my_checkpoint.ckp. jnltool.pl --to-cluster --server-id=ws1 --gzip-output my_checkpoint.ckp Configuration information is assumed to be found in a file called my_checkpoint.yaml in the same directory as the input checkpoint. Prerequisites You must satisfy the following prerequisites to use jnltool.pl. The checkpoint used for seeding must be created for a server or later. The checkpoint used for seeding must not be compressed. Reserve sufficient disk space: you need double the space of the source checkpoint to create a single node in the cluster. You must have Perl version 5.14 or later. Create a checkpoint You can create a checkpoint to serve as an input for the jnltool.pl tool by using one of the following commands: p4d -r server_root -jc directory or: p4 admin checkpoint directory Perforce Server Administrator's Guide: Cluster Management 23