THE ROUTE TO ROOTLESS

Transcription

1 THE ROUTE TO ROOTLESS

2 THE ROUTE TO ROOTLESS

3 BILL AND TED'S ROOTLESS ADVENTURE

4 THE ROUTE TO ROOTLESS

5 WHAT SECURITY PROBLEM IS GARDEN SOLVING IN CLOUD FOUNDRY?

6 THE PROBLEM IN CLOUD FOUNDRY Public Multi-Tenant Docker Workloads

7 WHAT IS A CONTAINER?

8 THE GREATEST TRICK CONTAINERS EVER PULLED WAS CONVINCING THE WORLD THEY EXIST

9 WHAT IS A CONTAINER? Confinement Own view of the system Fair share of resources Unable to modify constraints Dependency Management

10 CONFINEMENT Linux Namespaces Cgroups Dropping Capabilities Seccomp AppArmor

11 LINUX NAMESPACES There are global resources in linux such as process trees, mount tables, network devices, etc. Namespaces wrap these global system resources to make it appear to the processes within the namespace that they have their own isolated instance of the global resource.

12 LINUX NAMESPACES PID - Process IDs MNT - Mount points NET - Network devices, stacks, ports, etc UTS - Hostname and NIS domain name IPC - Inter Process Communication USER - User and group IDs CGROUP - Cgroup root directory

13 SHARING Control Groups Resource limiting Prioritization Accounting Control Disk Quotas More on this trainwreck later!

14 CAPABILITIES Historically, processes could be privileged (effective user ID = 0, known as root) or unprivileged. Privileged processes would bypass all kernel permission checks. Since 2.2 Linux has divided superuser permissions into distinct units known as capabilities, which can be independently enabled or disabled.

15 CAPABILITIES CAP_SET_UID (change uid) CAP_NET_BIND_SERVICE (listen on privileged ports) CAP_KILL (send signals to any process) CAP_CHOWN (chown any files) CAP_DAC_OVERRIDE (bypass permission checks) CAP_SYS_ADMIN (do all the things!)

16 SECCOMP Seccomp stands for secure computing mode. It's a kernel sandboxing tool since Linux version Enabling seccomp on a process limits the system calls available to that process. Also can limit args allowable in a system call (e.g. namespace clone flags).

17 APPARMOR AppArmor is a mandatory access control mechanism. Profiles are applied to running process to limit access to resources or privilege. rwklx`

18

19 I GET KNOCKED DOWN, BUT I GET UP AGAIN CVE : runc fd traversal: User Namespaces, Capability Dropping, AppArmor CVE : SCSI MICDROP - User Namespaces, AppArmor CVE : ebpf verifier vulnerability - Capability Dropping (sometimes), Seccomp

20 DEPENDENCY MANAGEMENT pivot_root(8) Layered filesystems

21 PIVOT ROOT What's in /? run.sh Boring Host Ubuntu Cool Container Busybox

22 PIVOT ROOT pivot_root! run.sh Boring Host Ubuntu Cool Container Busybox

23 PIVOT ROOT pivot_root! run.sh Boring Host Ubuntu Cool Container Busybox

24 PIVOT ROOT What's in /? run.sh Boring Host Ubuntu Cool Container Busybox

25 LAYERED FILESYSTEMS run.sh

26 LAYERED FILESYSTEMS run.sh /bin/os-specific

27 LAYERED FILESYSTEMS run.sh /bin/os-specific AMI

28 LAYERED FILESYSTEMS run.sh Δ B Δ A Base ROOTFS

29 WHAT IS A CONTAINER? Confinement Own view of the system Fair share of resources Unable to modify constraints Dependency Management

30 SO EVERYTHING IS SECURE IN A CLOUD FOUNDRY CONTAINER RIGHT?

31 YES...BUT...

32

33

34 ROUTE TO ROUTELESS

35 CREDITS Jessie Frazelle Aleksa Sarai Akihiro Suda

36 WHAT IS A CONTAINER? Confinement Own view of the system Fair share of resources Unable to modify constraints Dependency Management

37 CONFINEMENT Unprivileged user namespaces Since Linux 3.8 Just need CAP_SYS_ADMIN in the owning user namespace to do the rest: Other namespaces Seccomp AppArmor

38 HOW DO USER NAMESPACES WORK? Users are living a double life...

39 IN THE HOST

40 IN THE CONTAINER

41 HOW DO USER NAMESPACES WORK? /proc/self/uid_map and /proc/self/gid_map specify user id mappings from outer to inner user namespace. Mappings are triples of (inner ID, outer ID, range)

42 CONFINEMENT How do unprivileged user namespaces work? newuidmap/newgidmap setuid binaries validate mappings against /etc/subuid PRed support to runc

43 SHARING There's no way to do cgroups entirely unprivileged yet cgroups are a virtual filesystem like proc /sys/fs/cgroup/memory/** Files are owned by root by default Privileged setup can chown cgroups to our container root user, so runc can write to them PRed support to runc

44 DEPENDENCY MANAGEMENT pivot_root(8) User namespace gives us CAP_SYS_ADMIN

45 DEPENDENCY MANAGEMENT Layered filesystems AUFS mounts Not possible unprivileged BTRFS Snapshots OverlayFS mounts

46 DEPENDENCY MANAGEMENT Layered filesystems AUFS mounts (not possible unprivileged) BTRFS Snapshots Initial setup as privileged Snapshots can be done unprivileged Exploded with quotas at scale :( OverlayFS mounts (seems to be working?!)

47 DEPENDENCY MANAGEMENT Layered filesystems AUFS mounts (not possible unprivileged) BTRFS Snapshots OverlayFS mounts Possible on Ubuntu unprivileged Seems to be working?!

48 OVERLAY ROOTFUL

49 OVERLAY ROOTLESS

50 ROAD BLOCKS

51 DISK QUOTAS We use XFS for filesystem quotas XFS requires privilege Small, focused setuid binary just for quota management

52 NETWORKING Networking used to be integrated into Garden Garden supports a plugin architecture garden-external-networker is setuid Some awesome work going on from Aleksa Sarai and Akihiro Suda on this front!

53 GDN SETUP cgroup chowning in garden setup No user input Before any workload can be running in CF At least one attempt to fix this by Aleksa Sarai, but no luck yet

54 BUT IT'S OK...

55 REDUCING PRIVILEGE Reduce privilege where we can, when we can Break apart monoliths to only allow privilege where required Some things take time Proving out slowly is a positive thing It's getting better!

56 DOES IT WORK?

57 HOW CAN I TRY IT? Experimental config option in BOSH manifest

58 THANKS

59 "ET TU ROOT?" - JULIUS CAESAR