Systemd. Simone Caronni. University of Zurich 29 th September 2015

Transcription

1 Systemd Simone Caronni University of Zurich 29 th September 2015

2 What is systemd? Controls units rather than just daemons Handles dependency between units. Tracks processes with service information Services are owned by a cgroup. Simple to configure SLAs based on CPU, Memory, and IO Properly kill daemons, prevent forked processes from running stray Minimal boot times Debuggability no early boot messages are lost Easy to learn and backwards compatible Log aggregation

3 What does systemd replace? init udev pm-utils inetd acpid crond/atd ConsoleKit automount watchdog syslog (locally) The core components are always built (which includes systemd itself, as well as udevd and journald). Many of the other components can be disabled at compile time with configure switches.

4 Configuration files

5 Configuration files /etc/hostname: the host name for the system. All non-systemd distributions used different files for this. Fedora used /etc/sysconfig/network, OpenSUSE /etc/hostname. /etc/vconsole.conf: configuration of the default keyboard mapping and console font. /etc/locale.conf: configuration of the system-wide locale. /etc/modules-load.d/*.conf: a drop-in directory for kernel modules to statically load at boot. /etc/sysctl.d/*.conf: a drop-in directory for kernel sysctl parameters, extending what you can already do with /etc/sysctl.conf. /etc/tmpfiles.d/*.conf: a drop-in directory for configuration of runtime files that need to be removed/created/cleaned up at boot and during uptime.

6 Configuration files /etc/binfmt.d/*.conf: a drop-in directory for registration of additional binary formats for systems like Java, Mono and WINE. /etc/os-release: a standardization of the various distribution ID files like /etc/fedora-release and similar. Everybody can use the same file here. /etc/machine-id: a machine ID file, superseding D-Bus' machine ID file. This file is guaranteed to be existing and valid on a systemd system, covering also stateless boots. Originally in D-Bus, it is a unique and stable machine identifier. /etc/machine-info: a new information file encoding meta data about a host, like a pretty host name and an icon name, replacing stuff like /etc/favicon.png and suchlike. This is maintained by systemd-hostnamed.

7 Managing services and runlevels units, systemctl, troubleshooting

8 systemd units name.service name.socket name.device name.mount name.automount name.swap name.target name.path name.timer name.snapshot name.slice name.scope

9 httpd.service [Unit] Description=The Apache HTTP Server After=remote-fs.target nss-lookup.target [Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ExecReload=/usr/sbin/httpd $OPTIONS -k graceful ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop KillSignal=SIGCONT PrivateTmp=true [Install] WantedBy=multi-user.target

10 Managing services Init Init scripts: /etc/init.d & called from /etc/rc* systemd Maintainer files: /usr/lib/systemd/system Administrator files: /etc/systemd/system Non-persistent, runtime data: /run/systemd Note: unit files under /etc will take precedence over /usr

11 Managing services systemctl list-units systemctl enable/disable foo.service systemctl start/stop/restart foo.service systemctl reload foo.service systemctl status foo.service systemctl mask/unmask foo.service

12 Managing services Glob units when needed. systemctl restart httpd mariadb systemctl enable httpd mariadb ntpd lm_sensors [etc] If a unit type isn't specified,.service is assumed. systemctl start httpd == systemctl start httpd.service Shell completion is highly recommended Install bash-completion Add bash-completion to minimal kickstarts Connect to remote hosts over SSH using -H

13 Managing services

14 Managing services - systemctl List loaded services: systemctl -t service List installed services: systemctl list-unit-files -t service (similar to chkconfig --list) View state: systemctl --state failed

15 Managing runlevels Runlevels are exposed as target units Target names are more relevant: multi-user.target vs. runlevel3 graphical.target vs. runlevel5 View the default target: systemctl get-default Set the default target: systemctl set-default [target] Change at run-time: systemctl isolate [target] Note: /etc/inittab is no longer used.

16 Managing runlevels $ ls -ld /etc/systemd/system/default.target lrwxrwxrwx. 1 root root 36 Oct /etc/systemd/system/default.target -> /lib/systemd/system/runlevel5.target $ ls -ld /lib/systemd/system/runlevel5.target lrwxrwxrwx 1 root root 16 Jan 20 20:38 /lib/systemd/system/runlevel5.target -> graphical.target

17 /lib/systemd/system/graphical.target [Unit] Description=Graphical InterfaceDocumentation=man:systemd.special(7) Requires=multi-user.target After=multi-user.target Conflicts=rescue.target Wants=display-manager.service AllowIsolate=yes [Install] Alias=default.target

18 Troubleshooting Early boot shell on tty9 systemctl enable debug-shell.service Equivalent to: ln -s /usr/lib/systemd/system/debugshell.service /etc/systemd/system/sysinit.target.wants/ systemctl list-jobs systemd-analyze Use blame, plot, or critical-chain for more details

19 Troubleshooting systemd-analyze blame

20 Customizing units List a unit's properties: systemctl show --all httpd Query a single property: systemctl show -p Restart httpd Restart=no Helpful man files: systemd.exec and systemd.service Restart, Nice, CPUAffinity, OOMScoreAdjust, LimitNOFILE, etc Disclaimer: just because you can configure something doesn't mean you should!

21 Customizing units (drop-ins) mkdir /etc/systemd/system/httpd.service.d/ vim /etc/systemd/system/httpd.service.d/50- httpd.conf [Service] (Remember the 'S' is capitalized) Restart=always CPUAffinity= OOMScoreAdjust=-1000 systemctl daemon-reload Changes will be applied on top of maintainer unit files.

22 Customizing units (drop-ins)

23 Customizing units (drop-ins) systemd-delta is your friend. Simple to use with configuration tools like Satellite, Puppet, etc. Simply delete the drop-in to revert to defaults. Don't forget systemctl daemon-reload when modifying units.

24 Converting services SysV, inetd (socket activated services)

25 Converting init scripts Systemd maintains 99% backwards compatibility with LSB compatible initscripts and the exceptions are well documented. Users are encouraged to convert legacy scripts to service unit files, but it's not a requirement. Incompatibilities are listed here: / Converting SysV Init Scripts:

26 Converting inetd services SSH with classic inetd: ssh stream tcp nowait root /usr/sbin/sshd sshd -i SSH with xinetd: service ssh { socket_type = stream protocol = tcp wait = no user = root server = /usr/sbin/sshd server_args = -i }

27 Converting inetd services sshd.socket [Unit] Description=SSH Socket for Per-Connection Servers [Socket] ListenStream=22 Accept=yes [Install] WantedBy=sockets.target [Unit] Description=SSH Per-Connection Server [Service] ExecStart=-/usr/sbin/sshd -i StandardInput=socket

28 Converting inetd services is an instantiated service,for each incoming connection systemd will instantiate a new instance of sshd@.service, with the instance identifier named after the connection credentials. Example for running sessions: $ systemctl --full grep ssh sshd@ : :47779.service loaded active running SSH Per-Connection Server sshd@ : :52985.service loaded active running SSH Per-Connection Server sshd.socket loaded active listening SSH Socket for Per- Connection Servers

29 Resource management slices, scopes, services

30 Resource management Trivial to keep poorly written applications from stomping on your system. Configure how applications, containers, and VMs will behave when resources are under contention. Leverages the existing Control Groups implementation.

31 Resource management Slice Unit type for creating the cgroup hierarchy for resource management. Scope Organizational unit that groups a services' worker processes. Service Process or group of processes controlled by systemd Recursively show control group contents: systemd-clgs ps xawf -eo pid,user,cgroup,args Show top control groups by their resource usage systemd-cgtop

32 Understanding the hierarchy -/ Systemd implements a standard, single-root hierarchy under /sys/fs/cgroup/systemd

33 Understanding the hierarchy -/ user.slice system.slice machine.slice Each slice gets equal CPU time on the scheduler: CPUShares=1024

34 Understanding the hierarchy -/ user.slice system.slice machine.slice user-1.slice user-2.slice session.3-scope bash sshd:user

35 Understanding the hierarchy -/ user.slice system.slice machine.slice user-1.slice user-2.slice tomcat.service session.3-scope sshd.service bash mariadb.service sshd:user

36 Understanding the hierarchy -/ user.slice system.slice machine.slice user-1.slice user-2.slice tomcat.service vm1.scope session.3-scope sshd.service qemu bash mariadb.service vm2.scope sshd:user qemu

37 Understanding the hierarchy systemd-cgls

38 Understanding the hierarchy

39 Understanding the hierarchy systemd-cgtop

40 Resource management Enable the desired controller(s) CPU, Memory, BlockIO Configure cgroup attributes: systemctl set-property --runtime httpd.service CPUShares=2048 Drop --runtime to persist: systemctl set-property httpd.service CPUShares=2048 Or place in the unit file: [Service] CPUShares=2048

41 Resource management CPUAccounting=1 to enable CPUShares default is Increase to assign more CPU to a service e.g. CPUShares=1600

42 Resource management MemoryAccounting=1 to enable MemoryLimit= Use K, M, G, T suffixes, e.g. MemoryLimit=1G

43 Resource management BlockIOAccounting=1 BlockIOWeight= assigns an IO weight to a specific service (requires CFQ) Similar to CPU shares Default is 1000 Range Can be defined per device (or mount point) BlockIOReadBandwidth and BlockIOWriteBandwidth BlockIOWriteBandwith=/var/log 5M

44 The journal journalctl, journald

45 The journal Indexed Formatted Errors in red Warnings in bold Security Reliability Intelligently rotated By default, journal users can only watch their own logs, unless they are root or in the adm group. Systemd logs messages in the journal daemon coming in via /dev/log via the native protocol via STDOUT/STDERR of all services via the kernel.

46 The journal Does not replace rsyslog yet in RHEL 7 rsyslog is enabled by default The journal is not persistent by default. Enable persistence: mkdir /var/log/journal Stored in key-value pairs journalctl [tab] [tab] man 7 systemd.journal-fields Collects event metadata along with the message Simple to filter Interleave units, binaries, etc.

47 journalctl Page logs for one or more units journalctl -u foo Tail the journal journalctl -f Show X number of lines journalctl -n 50 View from boot journalctl -b

48 journalctl Other useful filters: -r reverse order binary e.g. /usr/sbin/dnsmasq [additional binaries] --since=yesterday or YYYY-MM-DD (HH:MM:SS) --until=yyyy-mm-dd View entire journal (useful for grep): journalctl -o verbose

49 journalctl Filter by priority: journalctl -p [level] 0 emerg 1 alert 2 crit 3 err 4 warning 5 notice 6 debug

50 journalctl Using system variables journalctl _SYSTEMD_UNIT=foo.service journalctl _UID=1000 _UID=1001 Journalctl _COMM=Xorg.bin journalctl _SELINUX_CONTEXT=system_u:system_r:policykit _t:s0 Combining search with OR journalctl _UID= _COMM=Xorg.bin Show us all values the field _SYSTEMD_UNIT takes in the database (the names of all systemd services which ever logged into the journal) journalctl -F _SYSTEMD_UNIT

51 syslog logging Systemd logs messages in the journal daemon coming in via /dev/log via the native protocol via STDOUT/STDERR of all services via the kernel. The journal daemon then stores them to disk or in RAM (depending on the configuration of the Storage= option in journald.conf), and optionally forwards them to the console, the kernel log buffer, or to a classic BSD syslog daemon. It is now the journal that listens on /dev/log, no longer the syslog daemon directly. If your logging daemon wants to get access to all logging data then it should listen on /run/systemd/journal/syslog instead via the syslog.socket unit file that is shipped along with systemd. Your logging service should alias syslog.service to itself (i.e. symlink) when it is enabled. That way syslog.socket will activate your service when things are logged. Sample logging daemon configuration file [Unit] Description=System Logging Service Requires=syslog.socket [Service] ExecStart=/usr/sbin/syslog-ng -n StandardOutput=null [Install] Alias=syslog.service WantedBy=multi-user.target journald tries hard to forward to your syslog daemon as much as it can. That means you will get more than you traditionally got on /dev/log, such as stuff all daemons log on STDOUT/STDERR and the messages that are logged natively to systemd.

52 Advanced features Additional service customizations Specific installations systemd-logind integration

53 systemd-logind systemd-logind is a system service that manages user logins Keeps track of users and sessions, their processes and their idle state Provides PolicyKit-based access for users to operations such as system shutdown or sleep Implements a shutdown/sleep inhibition logic for applications Handles of power/sleep hardware keys Does multi-seat management Does session switch management Does device access management for users Performs automatic spawning of text logins (gettys) on virtual console activation and user runtime directory management User sessions are registered in logind via the pam_systemd(8) PAM module.

54 Unprivileged X.org Can be run unprivileged, delegating device management to systemd-logind. It has an API to allow a process inside a session to become the session controller, and get filedescriptors for these device nodes over dbus. Can be made into a socket activated service For compatibility, if using binary drivers (nvidia, fglrx), the login manager and the X.org server run still as root.

55 Presets Different distributions have different policies on which services shall be enabled by default when the package they are shipped in is installed. On Fedora all services stay off by default, so that installing a package will not cause a service to be enabled (with some exceptions). On Debian all services are immediately enabled by default, so that installing a package will cause its service(s) to be enabled right-away. Different spins (flavours, remixes, whatever you might want to call them) of a distribution also have different policies on what services to enable, and what services to leave off. For example, the Fedora default will enable gdm as display manager by default, while the Fedora KDE spin will enable kdm instead. Different sites might also have different policies what to turn on by default and what to turn off. For example, one administrator would prefer to enforce the policy of "ssh should be always on, but everything else off", while another one might say "snmp always on, and for everything else use the distribution policy defaults". systemd supports package preset policies. These encode which units shall be enabled by default when they are installed, and which units shall not be enabled. Preset files may be written for specific distributions, for specific spins or for specific sites, in order to enforce different policies as needed. Preset policies are stored in.preset files in /usr/lib/systemd/system-preset/. If no policy exists the default implied policy of "enable everything" is enforced, i.e. in Debian style. The policy encoded in preset files is applied to a unit by invoking systemctl preset.

56 Multi-seat support A seat consists of all hardware devices assigned to a specific workplace. It consists of at least one graphics device, and usually also includes keyboard, mouse. It can also include video cameras, sound cards and more. Seats are identified by seat names that start with the four characters "seat" followed by some more characters. Seat names may or may not be stable and may be reused if a seat becomes available again. A session is defined by the time a user is logged in until he logs out. A session is bound to one or no seats (the latter for 'virtual' ssh logins). Multiple sessions can be attached to the same seat, but only one of them can be active, the others are in the background. A session is identified by a short string, and systemd ensures that audit sessions are identical to systemd sessions. The session identifier is also a short string suitable for inclusion in a file name. Session IDs are unique on the local machine and are never reused as long as the machine is online. A single user can have opened multiple sessions at the same time. All hardware devices that are eligible to being assigned to a seat, are assigned to one. A device can be assigned to only one seat at a time. If a device is not assigned to any particular other seat it is implicitly assigned to the special default seat called "seat0" which always exists. Assignment of hardware devices to seats is managed inside the udev database, via settings on the devices: Tag "seat": if set this device is eligible to be assigned to a seat. Tag "master-of-seat": if set this device is enough for a seat to be considered existent. This tag is usually set for the framebuffer device of graphics cards. A seat hence consists of an arbitrary number of devices marked with the "seat" tag, but (at least) one of these devices needs to be tagged with "master-of-seat" before the seat is actually considered to be around. Property "ID_SEAT": if set specifies the name of the seat a specific device is assigned to. Property "ID_AUTOSEAT": if set to "1" then this device automatically generates a new seat and independent seat, which is named after the path of the device. Property "ID_FOR_SEAT": when creating additional (manual) seats starting from a graphics device this is a good choice to name the seat after. It is created from the path of the device. A seat exists only and exclusively because a properly tagged device with the right ID_SEAT property exists. Besides udev rules there is no persistent data about seats stored on disk.

57 Serial consoles To make use of a serial console line, just add console=ttys0 to your kernel command line, and systemd will enable it. If more than one serial login prompt is needed or the kernel console should be redirected to a different terminal than the login prompt. To facilitate this it is sufficient to instantiate serial-getty@.service once for each serial port you want it to run on systemctl enable serialgetty@ttys2.service systemctl start serialgetty@ttys2.service

58 Detecting virtualization Currently, the the virtualization detection code in systemd can detect the following virtualization systems: Hardware virtualization (i.e. VMs): qemu kvm vmware microsoft oracle xen bochs Same-kernel virtualization (i.e. containers): chroot openvz lxc lxc-libvirt systemd-nspawn

59 Isolating services from the network [Service] ExecStart= PrivateNetwork=yes Network interfaces became unavailable to the processes, the only one they'll see is the loopback device "lo", but it is isolated from the real host loopback.

60 Service private /tmp [Service] ExecStart= PrivateTmp=yes... If enabled this option will ensure that the /tmp directory the service will see is private and isolated from the host system's /tmp. /tmp traditionally has been a shared space for all local services and users. Over the years it has been a major source of security problems for a multitude of services.

61 Read-Only or inaccessible directories [Service] ExecStart= InaccessibleDirectories=/home ReadOnlyDirectories=/var... If enabled this option will ensure that the /tmp directory the service will see is private and isolated from the host system's /tmp. /tmp traditionally has been a shared space for all local services and users. Over the years it has been a major source of security problems for a multitude of services.

62 Removing capabilities [Service] ExecStart= CapabilityBoundingSet=CAP_CHOWN CAP_KILL... In the example above only the CAP_CHOWN and CAP_KILL capabilities are retained by the service, and the service and any processes it might create have no chance to ever acquire any other capabilities again, not even via setuid binaries. The list of currently defined capabilities is available in capabilities(7). The ~ character the value assignment prefixed inverts the meaning of the option: instead of listing all capabalities the service will retain you may list the ones it will not retain.

63 No forking, limiting file creation [Service] ExecStart= LimitNPROC=1 LimitFSIZE=0... Resource Limits may be used to apply certain security limits on services being run. Two of them can be useful to disable certain OS features: RLIMIT_NPROC and RLIMIT_FSIZE may be used to disable forking and disable writing of any files with a size > 0:

64 Device Node Access [Service] ExecStart= DeviceAllow=/dev/null rw... This will limit access to /dev/null and only this device node, disallowing access to any other device nodes. The feature is implemented on top of the devices cgroup controller.

65 Watchdog guarded services [Service] WatchdogSec=30s Restart=on-failure StartLimitInterval=5min StartLimitBurst=4 StartLimitAction=reboot-force This service will automatically be restarted if it hasn't pinged the system manager for longer than 30s or if it fails otherwise. If it is restarted this way more often than 4 times in 5min action is taken and the system quickly rebooted, with all file systems being clean when it comes up again. To make use of the hardware watchdog set the RuntimeWatchdogSec= option in /etc/systemd/system.conf. It defaults to 0 (i.e. no hardware watchdog use). ShutdownWatchdogSec= is another option that can be configured in /etc/systemd/system.conf. It controls the watchdog interval to use during reboots. It defaults to 10min, and adds extra reliability to the system reboot logic: if a clean reboot is not possible and shutdown hangs, we rely on the watchdog hardware to reset the system abruptly, as extra safety net.

66 Extra stuff

67 Additional components man -k systemd systemd-nspawn(1) - Spawn a namespace container for debugging, testing and building systemd-networkd(8) - Network manager -onmetal-servers/ -names/ and-systemd-networkd/ systemd-timesyncd(8) - Network Time Synchronization busctl (1) - Introspect the bus - along with: Sd-bus: kdbus: bootctl (1) - Control the firmware and boot manager Gummiboot: