Red Hat Enterprise Linux 6

Transcription

1 Red Hat Enterprise Linux 6 Resource Management Guide Edition 6 Managing system resources on Red Hat Enterprise Linux 6 Last Updated:

2

3 Red Hat Enterprise Linux 6 Resource Management Guide Managing system resources on Red Hat Enterprise Linux 6 Edition 6 Milan Navrátil Red Hat Customer Content Services mnavrati@redhat.com Marie Doleželová Red Hat Customer Content Services mdolezel@redhat.com Peter Ondrejka Red Hat Customer Content Services Eva Majoršinová Red Hat Customer Content Services Martin Prpič Red Hat Product Security Rüdiger Landmann Red Hat Customer Content Services Douglas Silas Red Hat Customer Content Services

4 Legal Notice Copyright 2016 Red Hat, Inc. This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the United States and other countries. Java is a registered trademark of Oracle and/or its affiliates. XFS is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project. The OpenStack Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community. All other trademarks are the property of their respective owners. Abstract Managing system resources on Red Hat Enterprise Linux 6.

5 Table of Contents Table of Contents. CHAPTER INTRODUCTION TO... CONTROL GROUPS (CGROUPS) HOW CONTROL GROUPS ARE ORGANIZED 4 The Linux Process Model 4 The Cgroup Model RELATIONSHIPS BETWEEN SUBSYSTEMS, HIERARCHIES, CONTROL GROUPS AND TASKS 5 Rule 1 5 Rule 2 6 Rule 3 6 Rule IMPLICATIONS FOR RESOURCE MANAGEMENT 8. CHAPTER USING CONTROL GROUPS THE CGCONFIG SERVICE The /etc/cgconfig.conf File The /etc/cgconfig.d/ Directory CREATING A HIERARCHY AND ATTACHING SUBSYSTEMS 12 Alternative method ATTACHING SUBSYSTEMS TO, AND DETACHING THEM FROM, AN EXISTING HIERARCHY 14 Alternative method UNMOUNTING A HIERARCHY CREATING CONTROL GROUPS 15 Alternative method REMOVING CONTROL GROUPS SETTING PARAMETERS 17 Alternative method MOVING A PROCESS TO A CONTROL GROUP 18 Alternative method The cgred Service STARTING A PROCESS IN A CONTROL GROUP 21 Alternative method Starting a Service in a Control Group Process Behavior in the Root Control Group GENERATING THE /ETC/CGCONFIG.CONF FILE Blacklisting Parameters Whitelisting Parameters OBTAINING INFORMATION ABOUT CONTROL GROUPS Finding a Process Finding a Subsystem Finding Hierarchies Finding Control Groups Displaying Parameters of Control Groups UNLOADING CONTROL GROUPS USING THE NOTIFICATION API ADDITIONAL RESOURCES 28. CHAPTER SUBSYSTEMS AND TUNABLE PARAMETERS BLKIO Proportional Weight Division Tunable Parameters I/O Throttling Tunable Parameters blkio Common Tunable Parameters Example Usage CPU 35 1

6 Resource Management Guide CFS Tunable Parameters RT Tunable Parameters Example Usage 3.3. CPUACCT 3.4. CPUSET 3.5. DEVICES 3.6. FREEZER 3.7. MEMORY Example Usage 3.8. NET_CLS 3.9. NET_PRIO NS PERF_EVENT COMMON TUNABLE PARAMETERS ADDITIONAL RESOURCES CHAPTER CONTROL GROUP APPLICATION EXAMPLES PRIORITIZING DATABASE I/O PRIORITIZING NETWORK TRAFFIC PER-GROUP DIVISION OF CPU AND MEMORY RESOURCES 60 Alternative method 63. APPENDIX A.... REVISION HISTORY

7 Table of Contents 3

8 Resource Management Guide CHAPTER 1. INTRODUCTION TO CONTROL GROUPS (CGROUPS) Red Hat Enterprise Linux 6 provides a new kernel feature: control groups, which are called by their shorter name cgroups in this guide. Cgroups allow you to allocate resources such as CPU time, system memory, network bandwidth, or combinations of these resources among user-defined groups of tasks (processes) running on a system. You can monitor the cgroups you configure, deny cgroups access to certain resources, and even reconfigure your cgroups dynamically on a running system. The cgconfig (control group config) service can be configured to start up at boot time and reestablish your predefined cgroups, thus making them persistent across reboots. By using cgroups, system administrators gain fine-grained control over allocating, prioritizing, denying, managing, and monitoring system resources. Hardware resources can be appropriately divided up among tasks and users, increasing overall efficiency HOW CONTROL GROUPS ARE ORGANIZED Cgroups are organized hierarchically, like processes, and child cgroups inherit some of the attributes of their parents. However, there are differences between the two models. The Linux Process Model All processes on a Linux system are child processes of a common parent: the init process, which is executed by the kernel at boot time and starts other processes (which may in turn start child processes of their own). Because all processes descend from a single parent, the Linux process model is a single hierarchy, or tree. Additionally, every Linux process except init inherits the environment (such as the PATH variable) [1] and certain other attributes (such as open file descriptors) of its parent process. The Cgroup Model Cgroups are similar to processes in that: they are hierarchical, and child cgroups inherit certain attributes from their parent cgroup. The fundamental difference is that many different hierarchies of cgroups can exist simultaneously on a system. If the Linux process model is a single tree of processes, then the cgroup model is one or more separate, unconnected trees of tasks (i.e. processes). Multiple separate hierarchies of cgroups are necessary because each hierarchy is attached to one or more subsystems. A subsystem [2] represents a single resource, such as CPU time or memory. Red Hat Enterprise Linux 6 provides ten cgroup subsystems, listed below by name and function. Available Subsystems in Red Hat Enterprise Linux blkio this subsystem sets limits on input/output access to and from block devices such as physical drives (disk, solid state, or USB). cpu this subsystem uses the scheduler to provide cgroup tasks access to the CPU. cpuacct this subsystem generates automatic reports on CPU resources used by tasks in a cgroup. 4

9 CHAPTER 1. INTRODUCTION TO CONTROL GROUPS (CGROUPS) cpuset this subsystem assigns individual CPUs (on a multicore system) and memory nodes to tasks in a cgroup. devices this subsystem allows or denies access to devices by tasks in a cgroup. freezer this subsystem suspends or resumes tasks in a cgroup. memory this subsystem sets limits on memory use by tasks in a cgroup and generates automatic reports on memory resources used by those tasks. net_cls this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task. net_prio this subsystem provides a way to dynamically set the priority of network traffic per network interface. ns the namespace subsystem. perf_event this subsystem identifies cgroup membership of tasks and can be used for performance analysis. NOTE You may come across the term resource controller or simply controller in cgroup literature such as the man pages or kernel documentation. Both of these terms are synonymous with subsystem and arise from the fact that a subsystem typically schedules a resource or applies a limit to the cgroups in the hierarchy it is attached to. The definition of a subsystem (resource controller) is quite general: it is something that acts upon a group of tasks, i.e. processes RELATIONSHIPS BETWEEN SUBSYSTEMS, HIERARCHIES, CONTROL GROUPS AND TASKS Remember that system processes are called tasks in cgroup terminology. Here are a few simple rules governing the relationships between subsystems, hierarchies of cgroups, and tasks, along with explanations of the consequences of those rules. Rule 1 A single hierarchy can have one or more subsystems attached to it. As a consequence, the cpu and memory subsystems (or any number of subsystems) can be attached to a single hierarchy, as long as each one is not attached to any other hierarchy which has any other subsystems attached to it already (see Rule 2). 5

10 Resource Management Guide Figure 1.1. Rule 1 Rule 2 Any single subsystem (such as cpu) cannot be attached to more than one hierarchy if one of those hierarchies has a different subsystem attached to it already. As a consequence, the cpu subsystem can never be attached to two different hierarchies if one of those hierarchies already has the memory subsystem attached to it. However, a single subsystem can be attached to two hierarchies if both of those hierarchies have only that subsystem attached. Figure 1.2. Rule 2 The numbered bullets represent a time sequence in which the subsystems are attached. Rule 3 Each time a new hierarchy is created on the systems, all tasks on the system are initially members of 6

11 CHAPTER 1. INTRODUCTION TO CONTROL GROUPS (CGROUPS) the default cgroup of that hierarchy, which is known as the root cgroup. For any single hierarchy you create, each task on the system can be a member of exactly one cgroup in that hierarchy. A single task may be in multiple cgroups, as long as each of those cgroups is in a different hierarchy. As soon as a task becomes a member of a second cgroup in the same hierarchy, it is removed from the first cgroup in that hierarchy. At no time is a task ever in two different cgroups in the same hierarchy. As a consequence, if the cpu and memory subsystems are attached to a hierarchy named cpu_mem_cg, and the net_cls subsystem is attached to a hierarchy named net, then a running httpd process could be a member of any one cgroup in cpu_mem_cg, and any one cgroup in net. The cgroup in cpu_mem_cg that the httpd process is a member of might restrict its CPU time to half of that allotted to other processes, and limit its memory usage to a maximum of 1024 MB. Additionally, the cgroup in net that the httpd process is a member of might limit its transmission rate to 30 MB/s (megabytes per second). When the first hierarchy is created, every task on the system is a member of at least one cgroup: the root cgroup. When using cgroups, therefore, every system task is always in at least one cgroup. Figure 1.3. Rule 3 Rule 4 Any process (task) on the system which forks itself creates a child task. A child task automatically inherits the cgroup membership of its parent but can be moved to different cgroups as needed. Once forked, the parent and child processes are completely independent. As a consequence, consider the httpd task that is a member of the cgroup named half_cpu_1gb_max in the cpu_and_mem hierarchy, and a member of the cgroup trans_rate_30 in the net hierarchy. When that httpd process forks itself, its child process automatically becomes a member of the half_cpu_1gb_max cgroup, and the trans_rate_30 cgroup. It inherits the exact same cgroups its parent task belongs to. From that point forward, the parent and child tasks are completely independent of each other: changing the cgroups that one task belongs to does not affect the other. Neither will changing cgroups of a parent task affect any of its grandchildren in any way. To summarize: any child task always initially inherits memberships to the exact same cgroups as their parent task, but those memberships can be changed or removed later. 7

12 Resource Management Guide Figure 1.4. Rule 4 The numbered bullets represent a time sequence in which the task forks IMPLICATIONS FOR RESOURCE MANAGEMENT Because a task can belong to only a single cgroup in any one hierarchy, there is only one way that a task can be limited or affected by any single subsystem. This is logical: a feature, not a limitation. You can group several subsystems together so that they affect all tasks in a single hierarchy. Because cgroups in that hierarchy have different parameters set, those tasks will be affected differently. It may sometimes be necessary to refactor a hierarchy. An example would be removing a subsystem from a hierarchy that has several subsystems attached, and attaching it to a new, separate hierarchy. Conversely, if the need for splitting subsystems among separate hierarchies is reduced, you can remove a hierarchy and attach its subsystems to an existing one. The design allows for simple cgroup usage, such as setting a few parameters for specific tasks in a single hierarchy, such as one with just the cpu and memory subsystems attached. The design also allows for highly specific configuration: each task (process) on a system could be a member of each hierarchy, each of which has a single attached subsystem. Such a configuration would give the system administrator absolute control over all parameters for every single task. [1] The parent process is able to alter the environment before passing it to a child process. [2] You should be aware that subsystems are also called resource controllers, or simply controllers, in the libcgroup man pages and other documentation. 8

13 CHAPTER 2. USING CONTROL GROUPS CHAPTER 2. USING CONTROL GROUPS As explained in Chapter 3, Subsystems and Tunable Parameters, control groups and the subsystems to which they relate can be manipulated using shell commands and utilities. However, the easiest way to work with cgroups is to install the libcgroup package, which contains a number of cgroup-related command line utilities and their associated man pages. It is possible to mount hierarchies and set cgroup parameters (non-persistently) using shell commands and utilities available on any system. However, using the libcgroup-provided utilities simplifies the process and extends your capabilities. Therefore, this guide focuses on libcgroup commands throughout. In most cases, we have included the equivalent shell commands to help describe the underlying mechanism. However, we recommend that you use the libcgroup commands wherever practical. NOTE In order to use cgroups, first ensure the libcgroup package is installed on your system by running, as root: ~]# yum install libcgroup 2.1. THE CGCONFIG SERVICE The cgconfig service installed with the libcgroup package provides a convenient way to create hierarchies, attach subsystems to hierarchies, and manage cgroups within those hierarchies. It is recommended that you use cgconfig to manage hierarchies and cgroups on your system. The cgconfig service is not started by default on Red Hat Enterprise Linux 6. When you start the service with chkconfig, it reads the cgroup configuration file /etc/cgconfig.conf. Cgroups are therefore recreated from session to session and remain persistent. Depending on the contents of the configuration file, cgconfig can create hierarchies, mount necessary file systems, create cgroups, and set subsystem parameters for each group. The default /etc/cgconfig.conf file installed with the libcgroup package creates and mounts an individual hierarchy for each subsystem, and attaches the subsystems to these hierarchies. The cgconfig service also allows to create configuration files in the /etc/cgconfig.d/ directory and to invoke them from /etc/cgconfig.conf. If you stop the cgconfig service (with the service cgconfig stop command), it unmounts all the hierarchies that it mounted The /etc/cgconfig.conf File The /etc/cgconfig.conf file contains two major types of entries mount and group. Mount entries create and mount hierarchies as virtual file systems, and attach subsystems to those hierarchies. Mount entries are defined using the following syntax: mount { subsystem = /cgroup/hierarchy; The libcgroup package automatically creates a default /etc/cgconfig.conf file when it is installed. The default configuration file looks as follows: 9

14 Resource Management Guide mount { cpuset = /cgroup/cpuset; cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct; memory = /cgroup/memory; devices = /cgroup/devices; freezer = /cgroup/freezer; net_cls = /cgroup/net_cls; blkio = /cgroup/blkio; The subsystems listed in the above configuration are automatically mounted to their respective hierarchies under the /cgroup/ directory. It is recommended to use these default hierarchies for specifying control groups. However, in certain cases you may need to create hierarchies manually, for example when they were deleted before, or it is beneficial to have a single hierarchy for multiple subsystems (as in Section 4.3, Per-group Division of CPU and Memory Resources ). Note that multiple subsystems can be mounted to a single hierarchy, but each subsystem can be mounted only once. See Example 2.1, Creating a mount entry for an example of creating a hierarchy. Example 2.1. Creating a mount entry The following example creates a hierarchy for the cpuset subsystem: mount { cpuset = /cgroup/red; the equivalent of the shell commands: ~]# mkdir /cgroup/red ~]# mount -t cgroup -o cpuset red /cgroup/red Since each subsystem can be mounted only once, the above commands would fail if cpuset is already mounted. Group entries create cgroups and set subsystem parameters. Group entries are defined using the following syntax: group <name> { [<permissions>] <controller> { <param name> = <param value>; Note that the permissions section is optional. To define permissions for a group entry, use the following syntax: perm { task { uid = <task user>; 10

15 CHAPTER 2. USING CONTROL GROUPS gid = <task group>; admin { uid = <admin name>; gid = <admin group>; See Example 2.2, Creating a group entry for example usage: Example 2.2. Creating a group entry The following example creates a cgroup for SQL daemons, with permissions for users in the sqladmin group to add tasks to the cgroup and the root user to modify subsystem parameters: group daemons { cpuset { cpuset.mems = 0; cpuset.cpus = 0; group daemons/sql { perm { task { uid = root; gid = sqladmin; admin { uid = root; gid = root; cpuset { cpuset.mems = 0; cpuset.cpus = 0; When combined with the example of the mount entry in Example 2.1, Creating a mount entry, the equivalent shell commands are: ~]# mkdir -p /cgroup/red/daemons/sql ~]# chown root:root /cgroup/red/daemons/sql/* ~]# chown root:sqladmin /cgroup/red/daemons/sql/tasks ~]# echo $(cgget -n -v -r cpuset.mems /) > /cgroup/red/daemons/cpuset.mems ~]# echo $(cgget -n -v -r cpuset.cpus /) > /cgroup/red/daemons/cpuset.cpus ~]# echo 0 > /cgroup/red/daemons/sql/cpuset.mems ~]# echo 0 > /cgroup/red/daemons/sql/cpuset.cpus 11

16 Resource Management Guide NOTE You must restart the cgconfig service for the changes in the /etc/cgconfig.conf file to take effect. However, note that restarting this service causes the entire cgroup hierarchy to be rebuilt, which removes any previously existing cgroups (for example, any existing cgroups used by libvirtd). To restart the cgconfig service, use the following command: ~]# service cgconfig restart When you install the libcgroup package, a sample configuration file is written to /etc/cgconfig.conf. The hash symbols ('#') at the start of each line comment that line out and make it invisible to the cgconfig service The /etc/cgconfig.d/ Directory The /etc/cgconfig.d/ directory is reserved for storing configuration files for specific applications and use cases. These files should be created with the.conf suffix and adhere to the same syntax rules as /etc/cgconfig.conf. The cgconfig service first parses the /etc/cgconfig.conf file and then continues with files in the /etc/cgconfig.d/ directory. Note that the order of file parsing is not defined, because it does not make a difference provided that each configuration file is unique. Therefore, do not define the same group or template in multiple configuration files, otherwise they would interfere with each other. Storing specific configuration files in a separate directory makes them easily reusable. If an application is shipped with a dedicated configuration file, you can easily set up cgroups for the application just by copying its configuration file to /etc/cgconfig.d/ CREATING A HIERARCHY AND ATTACHING SUBSYSTEMS WARNING The following instructions, which cover creating a new hierarchy and attaching subsystems to it, assume that cgroups are not already configured on your system. In this case, these instructions will not affect the operation of the system. Changing the tunable parameters in a cgroup with tasks, however, can immediately affect those tasks. This guide alerts you the first time it illustrates changing a tunable cgroup parameter that can affect one or more tasks. On a system on which cgroups are already configured (either manually, or by the cgconfig service) these commands fail unless you first unmount existing hierarchies, which affects the operation of the system. Do not experiment with these instructions on production systems. To create a hierarchy and attach subsystems to it, edit the mount section of the /etc/cgconfig.conf file as root. Entries in the mount section have the following format: 12

17 CHAPTER 2. USING CONTROL GROUPS subsystem = /cgroup/hierarchy; When cgconfig next starts, it will create the hierarchy and attach the subsystems to it. The following example creates a hierarchy called cpu_and_mem and attaches the cpu, cpuset, cpuacct, and memory subsystems to it. mount { cpuset = /cgroup/cpu_and_mem; cpu = /cgroup/cpu_and_mem; cpuacct = /cgroup/cpu_and_mem; memory = /cgroup/cpu_and_mem; Alternative method You can also use shell commands and utilities to create hierarchies and attach subsystems to them. Create a mount point for the hierarchy as root. Include the name of the cgroup in the mount point: ~]# mkdir /cgroup/name For example: ~]# mkdir /cgroup/cpu_and_mem Next, use the mount command to mount the hierarchy and simultaneously attach one or more subsystems. For example: ~]# mount -t cgroup -o subsystems name /cgroup/name Where subsystems is a comma separated list of subsystems and name is the name of the hierarchy. Brief descriptions of all available subsystems are listed in Available Subsystems in Red Hat Enterprise Linux, and Chapter 3, Subsystems and Tunable Parameters provides a detailed reference. Example 2.3. Using the mount command to attach subsystems In this example, a directory named /cgroup/cpu_and_mem already exists and will serve as the mount point for the hierarchy that you create. Attach the cpu, cpuset, and memory subsystems to a hierarchy named cpu_and_mem, and mount the cpu_and_mem hierarchy on /cgroup/cpu_and_mem: ~]# mount -t cgroup -o cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem You can list all available subsystems along with their current mount points (i.e. where the hierarchy they are attached to is mounted) with the lssubsys [3] command: ~]# lssubsys -am cpu,cpuset,memory /cgroup/cpu_and_mem net_cls ns cpuacct 13

18 Resource Management Guide devices freezer blkio This output indicates that: the cpu, cpuset, and memory subsystems are attached to a hierarchy mounted on /cgroup/cpu_and_mem, and the net_cls, ns, cpuacct, devices, freezer, and blkio subsystems are as yet unattached to any hierarchy, as illustrated by the lack of a corresponding mount point ATTACHING SUBSYSTEMS TO, AND DETACHING THEM FROM, AN EXISTING HIERARCHY To add a subsystem to an existing hierarchy, detach it from an existing hierarchy, or move it to a different hierarchy, edit the mount section of the /etc/cgconfig.conf file as root, using the same syntax described in Section 2.2, Creating a Hierarchy and Attaching Subsystems. When cgconfig next starts, it will reorganize the subsystems according to the hierarchies that you specify. Alternative method To add an unattached subsystem to an existing hierarchy, remount the hierarchy. Include the extra subsystem in the mount command, together with the remount option. Example 2.4. Remounting a hierarchy to add a subsystem The lssubsys command shows cpu, cpuset, and memory subsystems attached to the cpu_and_mem hierarchy: ~]# lssubsys -am cpu,cpuset,memory /cgroup/cpu_and_mem net_cls ns cpuacct devices freezer blkio Remount the cpu_and_mem hierarchy, using the remount option, and include cpuacct in the list of subsystems: ~]# mount -t cgroup -o remount,cpu,cpuset,cpuacct,memory cpu_and_mem /cgroup/cpu_and_mem The lssubsys command now shows cpuacct attached to the cpu_and_mem hierarchy: ~]# lssubsys -am cpu,cpuacct,cpuset,memory /cgroup/cpu_and_mem net_cls ns 14

19 CHAPTER 2. USING CONTROL GROUPS devices freezer blkio Analogously, you can detach a subsystem from an existing hierarchy by remounting the hierarchy and omitting the subsystem name from the -o options. For example, to then detach the cpuacct subsystem, simply remount and omit it: ~]# mount -t cgroup -o remount,cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem 2.4. UNMOUNTING A HIERARCHY You can unmount a hierarchy of cgroups with the umount command: ~]# umount /cgroup/name For example: ~]# umount /cgroup/cpu_and_mem If the hierarchy is currently empty (that is, it contains only the root cgroup) the hierarchy is deactivated when it is unmounted. If the hierarchy contains any other cgroups, the hierarchy remains active in the kernel even though it is no longer mounted. To remove a hierarchy, ensure that all child cgroups are removed before you unmount the hierarchy, or use the cgclear command which can deactivate a hierarchy even when it is not empty refer to Section 2.12, Unloading Control Groups CREATING CONTROL GROUPS Use the cgcreate command to create cgroups. The syntax for cgcreate is: cgcreate -t uid:gid -a uid:gid -g subsystems:path where: -t (optional) specifies a user (by user ID, uid) and a group (by group ID, gid) to own the tasks pseudofile for this cgroup. This user can add tasks to the cgroup. NOTE Note that the only way to remove a task from a cgroup is to move it to a different cgroup. To move a task, the user has to have write access to the destination cgroup; write access to the source cgroup is not necessary. -a (optional) specifies a user (by user ID, uid) and a group (by group ID, gid) to own all pseudofiles other than tasks for this cgroup. This user can modify access of the tasks in this cgroup to system resources. 15

20 Resource Management Guide -g specifies the hierarchy in which the cgroup should be created, as a comma separated list of subsystems associated with hierarchies. If the subsystems in this list are in different hierarchies, the group is created in each of these hierarchies. The list of hierarchies is followed by a colon and the path to the child group relative to the hierarchy. Do not include the hierarchy mount point in the path. For example, the cgroup located in the directory /cgroup/cpu_and_mem/lab1/ is called just lab1 its path is already uniquely determined because there is at most one hierarchy for a given subsystem. Note also that the group is controlled by all the subsystems that exist in the hierarchies in which the cgroup is created, even though these subsystems have not been specified in the cgcreate command refer to Example 2.5, cgcreate usage. Because all cgroups in the same hierarchy have the same controllers, the child group has the same controllers as its parent. Example 2.5. cgcreate usage Consider a system where the cpu and memory subsystems are mounted together in the cpu_and_mem hierarchy, and the net_cls controller is mounted in a separate hierarchy called net. Run the following command: ~]# cgcreate -g cpu,net_cls:/test-subgroup The cgcreate command creates two groups named test-subgroup, one in the cpu_and_mem hierarchy and one in the net hierarchy. The test-subgroup group in the cpu_and_mem hierarchy is controlled by the memory subsystem, even though it was not specified in the cgcreate command. Alternative method To create a child of the cgroup directly, use the mkdir command: ~]# mkdir /cgroup/hierarchy/name/child_name For example: ~]# mkdir /cgroup/cpu_and_mem/group REMOVING CONTROL GROUPS Use cgdelete to remove cgroups. It has similar syntax as cgcreate. Run the following command: cgdelete subsystems:path where: subsystems is a comma separated list of subsystems. path is the path to the cgroup relative to the root of the hierarchy. For example: 16 ~]# cgdelete cpu,net_cls:/test-subgroup

21 CHAPTER 2. USING CONTROL GROUPS cgdelete can also recursively remove all subgroups with the option -r. When you delete a cgroup, all its tasks move to its parent group SETTING PARAMETERS Set subsystem parameters by running the cgset command from a user account with permission to modify the relevant cgroup. For example, if cpuset is mounted to /cgroup/cpu_and_mem/ and the /cgroup/cpu_and_mem/group1 subdirectory exists, specify the CPUs to which this group has access with the following command: cpu_and_mem]# cgset -r cpuset.cpus=0-1 group1 The syntax for cgset is: cgset -r parameter=value path_to_cgroup where: parameter is the parameter to be set, which corresponds to the file in the directory of the given cgroup. value is the value for the parameter. path_to_cgroup is the path to the cgroup relative to the root of the hierarchy. For example, to set the parameter of the root group (if the cpuacct subsystem is mounted to /cgroup/cpu_and_mem/), change to the /cgroup/cpu_and_mem/ directory, and run: cpu_and_mem]# cgset -r cpuacct.usage=0 / Alternatively, because. is relative to the root group (that is, the root group itself) you could also run: cpu_and_mem]# cgset -r cpuacct.usage=0. Note, however, that / is the preferred syntax. NOTE Only a small number of parameters can be set for the root group (such as the cpuacct.usage parameter shown in the examples above). This is because a root group owns all of the existing resources, therefore, it would make no sense to limit all existing processes by defining certain parameters, for example the cpuset.cpu parameter. To set the parameter of group1, which is a subgroup of the root group, run: cpu_and_mem]# cgset -r cpuacct.usage=0 group1 A trailing slash after the name of the group (for example, cpuacct.usage=0 group1/) is optional. 17

22 Resource Management Guide The values that you can set with cgset might depend on values set higher in a particular hierarchy. For example, if group1 is limited to use only CPU 0 on a system, you cannot set group1/subgroup1 to use CPUs 0 and 1, or to use only CPU 1. You can also use cgset to copy the parameters of one cgroup into another existing cgroup. For example: cpu_and_mem]# cgset --copy-from group1/ group2/ The syntax to copy parameters with cgset is: cgset --copy-from path_to_source_cgroup path_to_target_cgroup where: path_to_source_cgroup is the path to the cgroup whose parameters are to be copied, relative to the root group of the hierarchy. path_to_target_cgroup is the path to the destination cgroup, relative to the root group of the hierarchy. Ensure that any mandatory parameters for the various subsystems are set before you copy parameters from one group to another, or the command will fail. For more information on mandatory parameters, refer to Important. Alternative method To set parameters in a cgroup directly, insert values into the relevant subsystem pseudofile using the echo command. In the following example, the echo command inserts the value of 0-1 into the cpuset.cpus pseudofile of the cgroup group1: ~]# echo 0-1 > /cgroup/cpu_and_mem/group1/cpuset.cpus With this value in place, the tasks in this cgroup are restricted to CPUs 0 and 1 on the system MOVING A PROCESS TO A CONTROL GROUP Move a process into a cgroup by running the cgclassify command, for example: ~]# cgclassify -g cpu,memory:group The syntax for cgclassify is: cgclassify -g subsystems:path_to_cgroup pidlist where: subsystems is a comma separated list of subsystems, or * to launch the process in the hierarchies associated with all available subsystems. Note that if cgroups of the same name exist in multiple hierarchies, the -g option moves the processes in each of those groups. Ensure that the cgroup exists within each of the hierarchies whose subsystems you specify here. path_to_cgroup is the path to the cgroup within its hierarchies. 18

23 CHAPTER 2. USING CONTROL GROUPS pidlist is a space-separated list of process identifier (PIDs). If the -g option is not specified, cgclassify automatically searches the /etc/cgrules.conf file (see Section 2.8.1, The cgred Service ) and uses the first applicable configuration line. According to this line, cgclassify determines the hierarchies and cgroups to move the process under. Note that for the move to be successful, the destination hierarchies must exist. The subsystems specified in /etc/cgrules.conf also have to be properly configured for the corresponding hierarchy in /etc/cgconfig.conf. You can also add the --sticky option before the pid to keep any child processes in the same cgroup. If you do not set this option and the cgred service is running, child processes are allocated to cgroups based on the settings found in /etc/cgrules.conf. However, the parent process remains in the cgroup in which it was first started. Using cgclassify, you can move several processes simultaneously. For example, this command moves the processes with PIDs 1701 and 1138 into cgroup group1/: ~]# cgclassify -g cpu,memory:group Note that the PIDs to be moved are separated by spaces and that the specified groups should be in different hierarchies. Alternative method To move a process into a cgroup directly, write its PID to the tasks file of the cgroup. For example, to move a process with the PID 1701 into a cgroup at /cgroup/cpu_and_mem/group1/: ~]# echo 1701 > /cgroup/cpu_and_mem/group1/tasks The cgred Service Cgred is a service (which starts the cgrulesengd service) that moves tasks into cgroups according to parameters set in the /etc/cgrules.conf file. Entries in the /etc/cgrules.conf file can take one of these two forms: user subsystems control_group user:command subsystems control_group Replace user with a user name or a group name prefixed with the "@" character. Replace subsystems with a comma separated list of subsystem names, control_group represents a path to the cgroup, and command stands for a process name or a full command path of a process. For example: maria devices /usergroup/staff This entry specifies that any processes that belong to the user named maria access the devices subsystem according to the parameters specified in the /usergroup/staff cgroup. To associate particular commands with particular cgroups, add the command parameter, as follows: maria:ftp devices /usergroup/staff/ftp 19

24 Resource Management Guide The entry now specifies that when the user named maria uses the ftp command, the process is automatically moved to the /usergroup/staff/ftp cgroup in the hierarchy that contains the devices subsystem. Note, however, that the daemon moves the process to the cgroup only after the appropriate condition is fulfilled. Therefore, the ftp process might run for a short time in the wrong group. Furthermore, if the process quickly spawns children while in the wrong group, these children might not be moved. Entries in the /etc/cgrules.conf file can include the following extra indicates a group instead of an individual user. For are all users in the admins group. * represents "all". For example, * in the subsystem field represents all mounted subsystems. % represents an item that is the same as the item on the line above. For example, the entries specified in the /etc/cgrules.conf file can have the following devices % % The above configuration ensures that processes owned by the adminstaff and labstaff access the devices subsystem according to the limits set in the admingroup cgroup. Rules specified in /etc/cgrules.conf can be linked to templates configured either in the /etc/cgconfig.conf file or in configuration files stored in the /etc/cgconfig.d/ directory, allowing for flexible cgroup assignment and creation. For example, specify the following template in /etc/cgconfig.conf: template users/%g/%u { cpuacct{ cpu { cpu.shares = "1000"; Then use the users/%g/%u template in the third row of a /etc/cgrules.conf entry, which can look as follows: peter:ftp cpu users/%g/%u The %g and %u variables used above are automatically replaced with group and user name depending on the owner of the ftp process. If the process belongs to peter from the adminstaff group, the above path is translated to users/adminstaff/peter. The cgred service then searches for this directory, and if it does not exist, cgred creates it and assigns the process to users/adminstaff/peter/tasks. Note that template rules apply only to definitions of templates in configuration files, so even if "group users/adminstaff/peter" was defined in /etc/cgconfig.conf, it would be ignored in favor of "template users/%g/%u". There are several other variables that can be used for specifying cgroup paths in templates: 20

25 CHAPTER 2. USING CONTROL GROUPS %u is replaced with the name of the user who owns the current process. If name resolution fails, UID is used instead. %U is replaced with the UID of the specified user who owns the current process. %g is replaced with the name of the user group that owns the current process, or with the GID if name resolution fails. %G is replaced with the GID of the cgroup that owns the current process. %p is replaced with the name of the current process. PID is used in case of name resolution failure. %P is replaced with the PID of the current processes STARTING A PROCESS IN A CONTROL GROUP IMPORTANT Some subsystems have mandatory parameters that must be set before you can move a task into a cgroup which uses any of those subsystems. For example, before you move a task into a cgroup which uses the cpuset subsystem, the cpuset.cpus and cpuset.mems parameters must be defined for that cgroup. The examples in this section illustrate the correct syntax for the command, but only work on systems on which the relevant mandatory parameters have been set for any controllers used in the examples. If you have not already configured the relevant controllers, you cannot copy example commands directly from this section and expect them to work on your system. Refer to Chapter 3, Subsystems and Tunable Parameters for a description of which parameters are mandatory for given subsystems. Launch processes in a cgroup by running the cgexec command. For example, this command launches the firefox web browser within the group1 cgroup, subject to the limitations imposed on that group by the cpu subsystem: ~]# cgexec -g cpu:group1 firefox The syntax for cgexec is: cgexec -g subsystems:path_to_cgroup command arguments where: subsystems is a comma separated list of subsystems, or * to launch the process in the hierarchies associated with all available subsystems. Note that, as with cgset described in Section 2.7, Setting Parameters, if cgroups of the same name exist in multiple hierarchies, the -g option creates processes in each of those groups. Ensure that the cgroup exists within each of the hierarchies whose subsystems you specify here. path_to_cgroup is the path to the cgroup relative to the hierarchy. 21

26 Resource Management Guide command is the command to run. arguments are any arguments for the command. You can also add the --sticky option before the command to keep any child processes in the same cgroup. If you do not set this option and the cgred service is running, child processes will be allocated to cgroups based on the settings found in /etc/cgrules.conf. The process itself, however, will remain in the cgroup in which you started it. Alternative method When you start a new process, it inherits the group of its parent process. Therefore, an alternative method for starting a process in a particular cgroup is to move your shell process to that group (refer to Section 2.8, Moving a Process to a Control Group ), and then launch the process from that shell. For example: ~]# echo $$ > /cgroup/cpu_and_mem/group1/tasks ~]# firefox Note that after exiting firefox, your existing shell is still in the group1 cgroup. Therefore, an even better way would be: ~]# sh -c "echo \$$ > /cgroup/cpu_and_mem/group1/tasks && firefox" Starting a Service in a Control Group You can start certain services in a cgroup. Services that can be started in cgroups must: use a /etc/sysconfig/servicename file use the daemon() function from /etc/init.d/functions to start the service To make an eligible service start in a cgroup, edit its file in the /etc/sysconfig directory to include an entry in the form CGROUP_DAEMON="subsystem:control_group" where subsystem is a subsystem associated with a particular hierarchy, and control_group is a cgroup in that hierarchy. For example: CGROUP_DAEMON="cpuset:group1" If cpuset is mounted to /cgroup/cpu_and_mem/, the above configuration translates to /cgroup/cpu_and_mem/group Process Behavior in the Root Control Group Certain blkio and cpu configuration options affect processes (tasks) running in the root cgroup in a different way than those in a subgroup. Consider the following example: 1. Create two subgroups under one root group: /rootgroup/red/ and /rootgroup/blue/ 2. In each subgroup and in the root group, define the cpu.shares configuration option and set it to 1. In the scenario configured above, one process placed in each group (that is, one task in /rootgroup/tasks, /rootgroup/red/tasks and /rootgroup/blue/tasks) consumes 33.33% of the CPU:

27 CHAPTER 2. USING CONTROL GROUPS /rootgroup/ process: 33.33% /rootgroup/blue/ process: 33.33% /rootgroup/red/ process: 33.33% Any other processes placed in subgroups blue and red result in the 33.33% percent of the CPU assigned to that specific subgroup to be split among the multiple processes in that subgroup. However, multiple processes placed in the root group cause the CPU resource to be split per process, rather than per group. For example, if /rootgroup/ contains three processes, /rootgroup/red/ contains one process and /rootgroup/blue/ contains one process, and the cpu.shares option is set to 1 in all groups, the CPU resource is divided as follows: /rootgroup/ processes: 20% + 20% + 20% /rootgroup/blue/ process: 20% /rootgroup/red/ process: 20% Therefore, it is recommended to move all processes from the root group to a specific subgroup when using the blkio and cpu configuration options which divide an available resource based on a weight or a share (for example, cpu.shares or blkio.weight). To move all tasks from the root group into a specific subgroup, you can use the following commands: rootgroup]# cat tasks >> red/tasks rootgroup]# echo > tasks GENERATING THE /ETC/CGCONFIG.CONF FILE Configuration for the /etc/cgconfig.conf file can be generated from the current cgroup configuration using the cgsnapshot utility. This utility takes a snapshot of the current state of all subsystems and their cgroups and returns their configuration as it would appear in the /etc/cgconfig.conf file. Example 2.6, Using the cgsnapshot utility shows an example usage of the cgsnapshot utility. Example 2.6. Using the cgsnapshot utility Configure cgroups on the system using the following commands: ~]# mkdir /cgroup/cpu ~]# mount -t cgroup -o cpu cpu /cgroup/cpu ~]# mkdir /cgroup/cpu/lab1 ~]# mkdir /cgroup/cpu/lab2 ~]# echo 2 > /cgroup/cpu/lab1/cpu.shares ~]# echo 3 > /cgroup/cpu/lab2/cpu.shares ~]# echo > /cgroup/cpu/lab1/cpu.rt_period_us ~]# echo > /cgroup/cpu/lab1/cpu.rt_runtime_us ~]# mkdir /cgroup/cpuacct ~]# mount -t cgroup -o cpuacct cpuacct /cgroup/cpuacct The above commands mounted two subsystems and created two cgroups, for the cpu subsystem, with specific values for some of their parameters. Executing the cgsnapshot command (with the - s option and an empty /etc/cgsnapshot_blacklist.conf file [4] ) then produces the following output: 23

28 Resource Management Guide ~]$ cgsnapshot -s # Configuration file generated by cgsnapshot mount { cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct; group lab2 { cpu { group lab1 { cpu { cpu.rt_period_us=" "; cpu.rt_runtime_us="0"; cpu.shares="3"; cpu.rt_period_us=" "; cpu.rt_runtime_us=" "; cpu.shares="2"; The -s option used in the example above tells cgsnapshot to ignore all warnings in the output file caused by parameters not being defined in the blacklist or whitelist of the cgsnapshot utility. For more information on parameter blacklisting, refer to Section , Blacklisting Parameters. For more information on parameter whitelisting, refer to Section , Whitelisting Parameters. By default, the output generated by cgsnapshot is returned on the standard output. Use the -f option to specify a file to which the output should be redirected. For example: ~]$ cgsnapshot -f ~/test/cgconfig_test.conf WARNING When using the -f option, note that it overwrites any content in the file you specify. Therefore, it is recommended not to direct the output straight to the /etc/cgconfig.conf file. The cgsnapshot utility can also create configuration files per subsystem. By specifying the name of a subsystem, the output will consist of the corresponding configuration for that subsystem: ~]$ cgsnapshot cpuacct # Configuration file generated by cgsnapshot mount { cpuacct = /cgroup/cpuacct; 24

29 CHAPTER 2. USING CONTROL GROUPS Blacklisting Parameters The cgsnapshot utility allows parameter blacklisting. If a parameter is blacklisted, it does not appear in the output generated by cgsnapshot. By default, the /etc/cgsnapshot_blacklist.conf file is checked for blacklisted parameters. If a parameter is not present in the blacklist, the whitelist is checked. To specify a different blacklist, use the -b option. For example: ~]$ cgsnapshot -b ~/test/my_blacklist.conf Whitelisting Parameters The cgsnapshot utility also allows parameter whitelisting. If a parameter is whitelisted, it appears in the output generated by cgsnapshot. If a parameter is neither blacklisted or whitelisted, a warning appears informing of this: ~]$ cgsnapshot -f ~/test/cgconfig_test.conf WARNING: variable cpu.rt_period_us is neither blacklisted nor whitelisted WARNING: variable cpu.rt_runtime_us is neither blacklisted nor whitelisted By default, there is no whitelist configuration file. To specify which file to use as a whitelist, use the -w option. For example: ~]$ cgsnapshot -w ~/test/my_whitelist.conf Specifying the -t option tells cgsnapshot to generate a configuration with parameters from the whitelist only OBTAINING INFORMATION ABOUT CONTROL GROUPS There are several ways to find and monitor control groups, subsystems, and hierarchies configured on your system Finding a Process To find the cgroup to which a process belongs, run: ~]$ ps -O cgroup Or, if you know the PID for the process, run: ~]$ cat /proc/pid/cgroup where PID stands for a PID of the inspected process Finding a Subsystem To find out what subsystems are available in your kernel and how they are mounted together into hierarchies, run: ~]$ cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset

30 Resource Management Guide ns cpu cpuacct memory devices freezer net_cls blkio perf_event net_prio In the example output above, the hierarchy column lists IDs of the existing hierarchies on the system. Subsystems with the same hierarchy ID are attached to the same hierarchy. The num_cgroup column lists the number of existing cgroups in the hierarchy that uses a particular subsystem. The enabled column reports the value of 1 if a particular subsystem is enabled, or 0 if it is not. Or, to find the mount points of particular subsystems, run: ~]$ lssubsys -m subsystems where subsystems is a list of the subsystems in which you are interested. Note that the lssubsys -m command returns only the top-level mount point per each hierarchy Finding Hierarchies It is recommended that you mount hierarchies under the /cgroup/ directory. Assuming this is the case on your system, list or browse the contents of that directory to obtain a list of hierarchies. If the tree utility is installed on your system, run it to obtain an overview of all hierarchies and the cgroups within them: ~]$ tree /cgroup Finding Control Groups To list the cgroups on a system, run: ~]$ lscgroup You can restrict the output to a specific hierarchy by specifying a controller and path in the format controller:path. For example: ~]$ lscgroup cpuset:group1 lists only subgroups of the group1 cgroup in the hierarchy to which the cpuset subsystem is attached Displaying Parameters of Control Groups To display the parameters of specific cgroups, run: ~]$ cgget -r parameter list_of_cgroups 26