In the sections below, we shall see how easy it is to work with btrfs and specifically how to recover from disk out-of-spac e issues.

Transcription

1 About btrfs and Docker Btrfs (pronounc ed better-fs) is a highly advanc ed and fully stable Linux filesystem that offers many c ompelling and useful features like Copy-On-Write, snapshotting, Raiding, and thin provisioning. Btrfs is the default filesystem on some systems like S us E / S LES and is being used by some very large financial organiz ations today. Within Docker, support for Btrfs is available in the form of a storage driver as of 1.10 (both open source and commercially supported versions). While S us E / S LES include btrfs as the default filesystem, it can be installed on almost every other Linux flavor like Debian / Ubuntu and Red Hat / CentOS systems. With the problems we see in using devic emapper and questions as to devic emapper's maintainability given that OverlayFS seems to be gaining more focus and attention, btrfs seems like a very good replacement for devic emapper on Red Hat based systems as Doc ker's preferred storage driver. Working with btrfs In the sections below, we shall see how easy it is to work with btrfs and specifically how to recover from disk out-of-spac e issues. What we'll use here is a stock Ubuntu LTS system. We'll install Docker 1.11, the btrfs package, configure Docker to use btrfs as the storage driver. We will then simulate a disk out-of-space issue and then look at steps to recover from it without losing any data or applications running on the system. For the purposes of understanding all commands are typed out by hand, and it is urged that these manual steps are automated to make the entire rec overy proc ess more reliable. Btrfs Quickstart 1. Let's login to our Ubuntu box and check if btrfs is installed and available. S ince, we will be dealing with making changes to filesystems it will be convenient to run all commands as root (or use sudo to prefix all c ommands). btrfs The program 'btrfs' is currently not installed. You can install it by typing: apt-get install btrfs-tools Expec tedly, btrfs is not installed on this Ubuntu system 2. Let's install btrfs as shown in the output message above 1

2 apt-get install btrfs-tools Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: liblzo2-2 The following NEW packages will be installed: btrfs-tools liblzo2-2 0 upgraded, 2 newly installed, 0 to remove and 91 not upgraded. Need to get 380 kb of archives. After this operation, 2,692 kb of additional disk space will be used. Do you want to continue? [Y/n] Get:1 trusty-updates/main liblzo2-2 amd ubuntu1.1 [46.1 kb] Get:2 trusty-updates/main btrfs-tools amd ubuntu0.1 [334 kb] Fetched 380 kb in 0s (18.3 MB/s) Selecting previously unselected package liblzo2-2:amd64. (Reading database files and directories currently installed.) Preparing to unpack.../liblzo2-2_ ubuntu1.1_amd64.deb... Unpacking liblzo2-2:amd64 ( ubuntu1.1)... Selecting previously unselected package btrfs-tools. Preparing to unpack.../btrfs-tools_3.12-1ubuntu0.1_amd64.deb... Unpacking btrfs-tools (3.12-1ubuntu0.1)... Processing triggers for man-db ( ubuntu1)... Setting up liblzo2-2:amd64 ( ubuntu1.1)... Setting up btrfs-tools (3.12-1ubuntu0.1)... update-initramfs: deferring update (trigger activated) Processing triggers for libc-bin (2.19-0ubuntu6.6)... Processing triggers for initramfs-tools (0.103ubuntu4.2)... update-initramfs: Generating /boot/initrd.img generic 3. Now the btrfs c ommand should work: btrfs --version 4. Also, ensure that the current version of docker is installed. The simplest way to get docker up and running is curl -fssl [ " sh 5. Btrfs works with block devices, so we would need one. Let's see what all block devices are attached to this system, using the lsblk c ommand: lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk xvda1 202:1 0 8G 0 part / There are no unpartitioned block devices on this system. The only one is the one on which the OS is installed. We need to add another one. S ince this is an AWS instance, I'm attaching two EBS instances, each 4GB in siz e. 2

3 Running lsblk command now should reflect the new block devices added. lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk xvda1 202:1 0 8G 0 part / xvdx 202: G 0 disk xvdy 202: G 0 disk 6. We need to change docker to use btrfs as the storage driver. Currently (as the default on Ubuntu), docker is using extfs as indic ated by the output of docker info grep -i filesystem docker info grep Filesystem WARNING: No swap limit support Backing Filesystem: extfs S o, let's shut down docker before we make any changes using service docker stop. Note: This is assuming that this is a brand new docker installation and there are no images that need to be preserved. If there are, then those images would need to be either pushed to the hub / DTR or exported into tar files and backed up in some fashion. Getting Docker to use Btrfs 1. Let's format the first disk xvdx to use the btrfs filesystem. The wonderful aspect of this exercise is that btrfs behaves as a logical volume manager and as such there is no need to partition the disk or anything. We could partition the disk if we really wanted to, and not because btrfs needed it. We'll use mkfs.btrfs for this task. mkfs.btrfs -f /dev/xvdx WARNING! - IS EXPERIMENTAL WARNING! - see before using Turning ON incompat feature 'extref': increased hardlink limit per file to fs created label (null) on /dev/xvdx nodesize leafsize sectorsize 4096 size 4.00GiB 2. Now that the btrfs filesystem has been created, let's mount it on the same path that docker uses as the root for all containers, images and volumes on the host which is /var/lib/docker. 3

4 df -lkh udev 3.9G 12K 3.9G 1% /dev tmpfs 799M 392K 799M 1% /run /dev/xvda1 7.8G 3.1G 4.3G 43% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 3.9G 0 3.9G 0% /run/shm none 100M 0 100M 0% /run/user mount /dev/xvdx /var/lib/docker df -lkh udev 3.9G 12K 3.9G 1% /dev tmpfs 799M 392K 799M 1% /run /dev/xvda1 7.8G 3.1G 4.3G 43% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 3.9G 0 3.9G 0% /run/shm none 100M 0 100M 0% /run/user /dev/xvdx 4.0G 320K 3.6G 1% /var/lib/docker 3. Running docker info after bringing the docker daemon back up should reflect that docker is indeed using the btrfs (as the) storage driver. service docker start docker start/running, process docker info grep -i storage WARNING: No swap limit support Storage Driver: btrfs Managing Disk Space Issues and Performance 1. Occasionally, we can expect to face 'out of disk space' issues, especially when dealing with big images or with images that have multiple tags and a CI / CD or automation system that could overwhelm the amount of available disk space. Handling disk space issues in Btrfs is actually quite trivial and it does not even require a restart of anything. Extending a volume with an additional device is as easy as running a single command on btrfs. In this section, we'll simulate conditions for an out of disk space condition and use btrfs to address it and see that docker continues to function with no hiccups 2. Let's simulate some very large Doc ker c ontainers using the fallocate c ommand: 4

5 docker run -it --name big-fat-container ubuntu bash Unable to find image 'ubuntu:latest' locally latest: Pulling from library/ubuntu f069f1d21059: Pull complete ecbeec5633cf: Pull complete ea6f18256d63: Pull complete 54bde7b02897: Pull complete Digest: sha256:bbfd93a02a8487edb60f20316ebc966ddc7aa123c2e b Status: Downloaded newer image for ubuntu:latest fallocate -l 3.4G simulation.ftty The output of df -lkh should reflect that the device has reached its storage threshold: df -lkh /var/lib/docker /dev/xvdx 4.0G 3.6G 61M 99% /var/lib/docker 3. Let's run another container, it would fail, expectedly as there is not enough disk space docker run -d tomcat Unable to find image 'tomcat:latest' locally latest: Pulling from library/tomcat 5c90d4a2d1a8: Downloading [==================================================>] MB/51.35 MB ab30c63719b1: Download complete be275827e8b7: Download complete 9aa4ff75c34e: Download complete a30607f3daa1: Downloading [> ] MB/77.64 MB ba18b6: Download complete 01a8aa3698c9: Download complete 7e5d5c1983f4: Downloading [==================================================>] MB/3.016 MB b9e603623f64: Waiting f43c0: Waiting ceed0: Waiting docker: write /var/lib/docker/tmp/getimageblob : no space left on device. See 'docker run --help'. 4. Btrfs' Logical Volume Management to the rescue. It is quite trivial now to easily expand our docker base directory (or btrfs volume, aptly) across additional, multiple disks. All it takes is a single command and an available block device to extend the volume upon. When we createdxvdx, we also created an additional EBS device xvdy. We will now use this unformatted block device to extend the existing /var/lib/docker volume, as shown below. Once we do this, it is recommended to "balance" the data across the multiple disks to offer resilienc e and improve performanc e. btrfs device add /dev/xvdy /var/lib/docker/ df -lkh /var/lib/docker /dev/xvdx 8.0G 3.6G 4.1G 47% /var/lib/docker 5

6 The output of df -lkh shows only 47% used as opposed to 99% used earlier when the volume was mounted on a single device. Matter of fact, when using Btrfs, the usual unix commands dfetc would not show accurate information due to the way Btrfs uses RAID. Therefore, a more accurate representation of disk free spac e c an be derived using btrfs filesystem df /var/lib/docker. The filesystem in the c ommand may be shortened to fi as shown below: btrfs fi df /var/lib/docker Data, single: total=3.57gib, used=3.51gib System, DUP: total=8.00mib, used=16.00kib System, single: total=4.00mib, used=0.00 Metadata, DUP: total=204.75mib, used=10.95mib Metadata, single: total=8.00mib, used=0.00 Now, new images can be pulled and we can run containers without any disk space issues: docker run -d tomcat Unable to find image 'tomcat:latest' locally latest: Pulling from library/tomcat 5c90d4a2d1a8: Pull complete ab30c63719b1: Pull complete be275827e8b7: Pull complete 9aa4ff75c34e: Pull complete a30607f3daa1: Pull complete ba18b6: Pull complete 01a8aa3698c9: Pull complete 7e5d5c1983f4: Pull complete b9e603623f64: Pull complete f43c0: Pull complete ceed0: Pull complete Digest: sha256:e7eb5e6620f88c94d bd1698c73a2d9f91fca6d5dd4a4322bfde2379 Status: Downloaded newer image for tomcat:latest 33ee2f0dd dd90c498b6ba559634d3329e63b81f6a03f9b6a6c4e As mentioned earlier, for better performance and resilience, it is a recommended practice that the disks be balanced. We can see that the disks are not balanced as most of the data was written to the first disk only. This is shown using btrfs fi show /var/lib/docker btrfs fi show /var/lib/docker Label: none uuid: c6542f8f-66b6-45da c21d4956a6 Total devices 2 FS bytes used 3.89GiB devid 1 size 4.00GiB used 4.00GiB path /dev/xvdx devid 2 size 4.00GiB used MiB path /dev/xvdy In the output above, all 4GiB of data has been written to /dev/xvdx while only about 832 MiB of space is used on /dev/xvdy The balancing of the data can be effected using the btrfs balance subcommand as shown below: 6

7 btrfs balance start /var/lib/docker Done, had to relocate 14 out of 14 chunks btrfs fi show /var/lib/docker Label: none uuid: c6542f8f-66b6-45da c21d4956a6 Total devices 2 FS bytes used 3.89GiB devid 1 size 4.00GiB used 1.91GiB path /dev/xvdx devid 2 size 4.00GiB used 2.72GiB path /dev/xvdy Summary We were able to see how Btrfs' powerful features allows us to manage disk space issues and improve performanc e and resilienc y through the inbuilt RAID support. BTRFS is an advanc ed and future generation file system and fits very well with Docker's way of working. S ince Btrfs treats all disks as a logical volume, it is trivial to extend volumes using a single command and it takes effect without the need to restart docker or make any other c onfiguration c hanges. There are some excellent and more indepth articles at Docker and Btrfs in practice ( s.doc ker.c om/engine/userguide/storagedriver/btrfs-driver/) on doc s.doc ker.c om. 7