Open collaboration in the Moby Project

Transcription

1 Open Source Forum (Nov 15, 2017) Open collaboration in the Moby Project Akihiro Suda ) NTT Software Innovation Center

2 Who am I Software Engineer at NTT Docker Moby core maintainer (github.com/docker/docker moby/moby) In April 2017, Docker [ as a project ] transited into Moby. Now Docker [ as a product ] has been developed as one of downstreams of Moby. : : RHEL Fedora 2

3 Who am I BuildKit initial maintainer (github.com/moby/buildkit) Next-generation `docker build` Executes DAG vertices of Dockerfile-equivalent concurrently Soon: cache-aware distributed mode containerd maintainer (github.com/containerd/containerd) Industry-standard container runtime Can be used as a Docker-replacement for Kubernetes 3

4 Agenda What is the Moby Project Recent collaborative work in the Moby Project Governance of the Moby Project Case study: How I became a maintainer (and how you can!) 4

5 What is the Moby Project? 5

6 What is the Moby Project? Anyone interested in adding "our Moby Project" to Wikipedia? 6

7 What is the Moby Project? A collaborative project that provides the "lego set" that can be assembled into container-based systems BuildKit Moby registry libentitlement DataKit VPNKit HyperKit Moby tool Moby engine (dockerd) libnetwork runc projects under as of Nov. 9. 7

8 What is the Moby Project? 9,149 Contributors 8,800 Pull Requests / Year Maintainers' affiliations: ADP Atomic Axway Cloudflare Codeship CoreOS Docker Dropbox Facebook Google Hamburger Software Huawei IBM InfluxData InfoSiftr Infoblox Kiratech Microsoft NTT Rancher Labs Red Hat SUSE Suranaree Univ. of Tech. Tencent vente-privee Anyone is eligible! Affiliations of the maintainers (including curators) of the projects under as of Nov. 9. Maybe not 100% accurate :P 8

9 Origin of the Moby Project Docker has been developed collaboratively by Docker, Inc. and contributors from various affiliations But Docker, Inc. has their own product roadmap In early 2017, Docker, Inc. and the community decided splitting the Docker Project from the Docker Product, so as to encourage more open collaboration Inspired by the Fedora community In April 2017, the Moby Project was established as the successor of the Docker Project 9

10 Relationship between Moby and its downstreams Similarity + Support, GUI.. Docker, Inc.'s products Other downstreams Docker Community Edition Docker Enterprise Edition Balena: Moby-based container engine for IoT, by Resin.io? Add your own downstream here... 10

11 Moby is not Docker `dockerd` command (daemon) is part of Moby Will be renamed to `moby-engine` soon `docker` command (CLI) is NOT part of Moby Because UX is basically out of scope of Moby Solely maintained by Docker, Inc. But still opensource. Mac/Win and cloud installers are NOT part of Moby Docker, Inc. 's proprietary software at the moment Dockerfile is being removed from Moby... But no worry! Moby BuildKit provides a new low-level instruction set

12 Recent collaborative work in the Moby Project Announcement at DockerCon EU (October 17, 2017) Kubernetes Docker Transition Docker API translator (unreleased) Kubernetes Container Runtime Interface (CRI) containerd containerd 12

13 Recent collaborative work in the Moby Project This was not a surprise, because Docker, Inc. and the Moby community has been already collaboratively working on Kubernetes-related stuff for a year 13

14 Recent collaborative work in the Moby Project In addition to support for Kubernetes, the Moby community is going to decouple more components People can work on their own respective stuff tactically, while collaborating on the common Moby "lego set" Place for open collaboration Respective work Examples: containerd, LinuxKit, InfraKit, BuildKit.. 14

15 containerd: Industry-standard container runtime Simpler architecture than the "monolith" of Docker Set of decoupled subsystems More collaborative than the past Docker in the pre-moby era Donated to CNCF Can be used as a Docker-replacement for Kubernetes (and so on) CRI-containerd, the glue module for k8s is mainly maintained by Google employees (k8s incubator) Kubernetes Docker (Moby) containerd v0.2 runc Transition Docker (Moby) Docker 18.XX (unreleased) Kubernetes CRI-containerd containerd v1.0 runc 15

16 containerd: Industry-standard container runtime containerd provides "a la carte" of well-decoupled subsystems Developers of containerd-based system can chose only what they want They can add their own custom plugins, while collaborating on the common plugins Docker registry OverlayFS Prometheus OCI Runtime OCI Image tarball btrfs Place for open collaboration Assemble OCI Image tarball Custom FS plugin OCI Runtime Home-made container system 16

17 LinuxKit: Toolkit for custom Linux distribution Assembles VM images for running certain service like Docker / Kubernetes Usecases are not limited to Docker / Kubernetes LinuxKit Docker Kubernetes LinuxKit Hyperkit Installer for mac Docker, Inc.'s Docker for Mac Place for open collaboration Redis LinuxKit EC2 Home-made "RedisOS" 17

18 InfraKit: Toolkit for infrastructure orchestration Deploys distributed system like Docker Swarm-mode to IaaS like EC2 Self-healing Recreate instances if the actual state differs from the desired state Now supports deploying Kubernetes as well Yuji Oshima (my colleague at NTT) implemented this with the community Through this collaboration, he became an InfraKit maintainer Docker Swarm-mode / Kubernetes LinuxKit EC2 LinuxKit EC2 LinuxKit EC2 Infrakit 18

19 BuildKit: Next-generation `docker build` Provides DAG-based low-level build instructions (LLB) DAG: Directed Acyclic Graph DAG allows running LLB instructions concurrently Cache-aware distributed mode (on Kubernetes) is on plan image://gcc git://foo/bar image://alpine Run("make") Run("apk add..") Image 19

20 BuildKit: Next-generation `docker build` LLB is expected to be compiled from human-friendly languages such as Dockerfile Dockerfile is going to be officially under Docker, Inc.'s control People can invent Dockerfile-alternative languages, while collaborating on the common LLB spec and toolkit (Docker, Inc.'s) Dockerfile Other languages Respective work Compile BuildKit LLB Place for open collaboration 20

21 BuildKit: next-generation `docker build` Even useful for non-container usecases, as a generic concurrent / distributed build toolkit We need more usecase and design feedback from the community 21

22 Moby governance Previously, Solomon Hykes (CTO, Docker, Inc.) was the BDFL Benevolent Dictator For Life On November 13, Technical Steering Committee (TSC) was established as the replacement for the BDFL role Elected by maintainers A single company cannot hold more than 1/3 seats 7 members Docker x2, Codeship, IBM, InfoSiftr, SEAL Systems, vente-privee ( ) 22

23 Moby Project, CNCF, and OCI Some components under the Moby umbrella belong to other organizations as well Moby TSC will help cross-project and cross-organization collaboration Moby Project OCI Runtime Spec Image Spec... Moby engine runc... CNCF Kubernetes... Overlapping projects under as of Nov

24 OCI: Open Containers Initiative Established in 2015 under the Linux Foundation umbrella to create the vendor-neutral container standards by Docker, CoreOS, and others In July 2017, OCI Runtime Spec v1.0 & Image Spec v1.0 were announced Moby projects implement OCI specs; so they can be easily integrated with other container projects runc (the reference OCI runtime) is governed by OCI, but still considered to belong to the Moby Project 24

25 OCI: Open Containers Initiative Currently, image distribution spec is out of the scope of OCI But many OCI folks are interested in bring the distribution spec into the OCI scope Docker Registry implementation (and spec?) is being moved to under the Moby Project Likely to have some influence on OCI, but still unclear :P 25

26 CNCF: Cloud Native Computing Foundation Established in 2015 under the Linux Foundation umbrella as well for hosting Kubernetes (Now 14 projects) containerd and Notary are CNCF projects but still belong to the Moby Project CNCF and k8s host some competing projects as well, but they are collaboratively implementing the common spec (OCI & k8s CRI: Container Runtime Interface) CRI-O (k8s incubator) rkt (CNCF) and rktlet (k8s incubator) 26

27 Landscape 2018 New lang? Moby BuildKit LLB Dockerfile compiler New CLI? Docker CLI Moby Engine API Part of Moby? Proprietary? Moby engine? k8s CRI Kubernetes BuildKit CRI-containerd OCI Image Spec Docker Registry API ( Moby? OCI?) containerd CRI-O OCI Runtime Spec runc 27

28 Who are maintainers and how they are elected Maintainers (aka "committers") can: Approve other contributors' pull requests (with 2 LGTMs usually) Manage GitHub issues Elect TSC members Add and remove other maintainers (with 66% approval vote) Maintainers are elected from active contributors who: Send pull requests (Bug fix, Enhancement, New feature...) Review other contributors pull requests Triage GitHub issues Anyone is eligible! 28

29 Case study: How I became a Docker/Moby maintainer Began contribution to Docker in December, 2015 Motivation: I was working on some fault injection tool (github.com/osrg/namazu), and got stuck in "false bug" of Dockerized ZooKeeper due to AUFS hang-up, and needed to look into the AUFS issue: #18180 This is not a bug of Docker but mainly tracked in Docker community (AUFS maintainer fixed that issue) Became a Docker maintainer in November, 2016 Docker Moby in April, 2017 (I think) Mainly contributed to filesystem issues Both AUFS and overlayfs have some stability and compatibility issues Also proposed some new features `docker network prune` (merged): #27525 introspection mount (procfs-like stuff for containers. unmerged yet): #24893, #26331 TCP port forwarder (withdrawn): #

30 Case study: How I became a BuildKit maintainer Initial maintainer from the beginning of the project (2017 summer) I proposed DAG-based builder (but without idea of LLB) Coincidently, Tõnis Tiigi (Docker, Inc.) was planning similar but even better idea, which turned into BuildKit Tõnis invited me to an initial maintainer of BuildKit 30

31

32

33 Case study: How I became a containerd maintainer Began contribution in December, 2016 Became a maintainer in September, 2017 Mainly contributed to filesystem and image issues Though contribution to Docker / Moby, I found filesystem issues are hard to maintain I needed to reform containerd interfaces and data formats for my experimental OCI Image extension FILEgrain: transport-agnostic, fine-grained content-addressable container image layout (github.com/akihirosuda/filegrain) 33

34 And how you can become a maintainer! (my personal view) Collaborativeness (the most important) "Best way to became an #opensource maintainer is to start helping like one" Comprehensiveness Issue analysis, Bug-fix, enhancement, feature addition... But when you plan to add a big feature, please coordinate with maintainers in GitHub issue or in Slack before opening a PR! Continuity One-shot contribution is always welcomed, but maintainership requires continuous activity But no need to inflate the numbers of your git commits 34

35 Good chances to contribution (my personal view) Moby engine Recently integrated containerd v1.0 runtime. We need to make sure there is no regression. The next step is to integrate containerd v1.0 snapshot subsystem. (Much harder for compatibility) BuildKit Docker compatibility is not stable. Testing is highly welcomed. Design for distributed mode is still under discussion. containerd Testing and performance optimization are welcomed. And more! 35

36 Recap The Moby Project is getting more collaborative You can contribute and become a maintainer 36