Reproducible Software Deployment with GNU Guix

Transcription

1 Reproducible Software Deployment with GNU Guix Ludovic Courtès Inria Rennes Bretagne Atlantique, November 2015

2 The difficulty of keeping software environments under control.

3 #1. Upgrades are hard.

4

5

6 #2. Stateful system management is intractable.

7 $DISTRO $DISTRO

8 $DISTRO apt-get update $DISTRO apt-get update state 1 a state 1 b

9 $DISTRO apt-get update $DISTRO apt-get update state 1 a state 1 b apt-get install foo apt-get remove bar state 2 a state 2 b

10 $DISTRO apt-get update $DISTRO apt-get update state 1 a state 1 b apt-get install foo apt-get remove bar state 2 a apt-get remove bar state 2 b apt-get install foo state 3 a state 3 b

11 $DISTRO apt-get update $DISTRO apt-get update state 1 a state 1 b state 2 a apt-get install foo =? apt-get remove bar state 2 b apt-get remove bar apt-get install foo state 3 a state 3 b

12 #3. It s worse than this.

13

14 It s worse, really.

15 Let s Package jquery: A Javascript Packaging Dystopian Novella by Chris Webber javascript-packaging-dystopia/

16 Giving up?

17 Giving up? app bundles (Docker images)

18

19

20 Functional package management.

21 openmpi = f (hwloc, gcc, make, coreutils) where f =./configure && make && make install

22 openmpi = f (hwloc, gcc, make, coreutils) hwloc = g(pciaccess, gcc, make, coreutils)

23 openmpi = f (hwloc, gcc, make, coreutils) hwloc = g(pciaccess, gcc, make, coreutils) gcc = h(make, coreutils, gcc 0 )...

24 openmpi = f (hwloc, gcc, make, coreutils) hwloc = g(pciaccess, gcc, make, coreutils) gcc = h(make, coreutils, gcc 0 )... the complete DAG is captured

25 A Safe and Policy-Free System for Software Deployment, Dolstra et al., 2003 Nix, Functional Package Management with Guix, Courtès, 2013

26 (define hello (package (name "hello") (version "2.10") (source (origin (method url-fetch) (uri (string-append "mirror://gnu/.../hello-" version ".tar.gz")) (sha256 (base32 "0wqd...dz6")))) (build-system gnu-build-system) (synopsis "Hello, world!") (description "Produce a friendly greeting.") (home-page " (license gpl3+)))

27 build processes chroot, separate UIDs Guile, make, etc. Guile, make, etc. Guile, make, etc. Guile Scheme (guix packages) (guix store) build daemon

28 build processes chroot, separate UIDs Guile, make, etc. Guile, make, etc. Guile, make, etc. Guile Scheme (guix packages) (guix store) build daemon RPCs

29 build processes chroot, separate UIDs Guile, make, etc. Guile, make, etc. Guile, make, etc. Guile Scheme (guix packages) (guix store) build daemon RPCs

30 $ guix build hello isolated build: chroot, separate name spaces, etc.

31 $ guix build hello /gnu/store/ h2g4sf hello-2.10 hash of all the dependencies

32 $ guix build hello /gnu/store/ h2g4sf hello-2.10 $ guix gc --references /gnu/store/...-hello-2.10 /gnu/store/...-glibc-2.22 /gnu/store/...-gcc lib /gnu/store/...-hello-2.10

33 $ guix build hello /gnu/store/ h2g4sf hello-2.10 $ guix gc --references /gnu/store/...-hello-2.10 /gnu/store/...-glibc-2.22 /gnu/store/...-gcc lib /gnu/store/...-hello-2.10 (nearly) bit-identical for everyone

34 $ guix package -i gcc-toolchain coreutils sed grep... demo $ eval guix package --search-paths... $ guix package --manifest=my-software.scm...

35 Want your PhD student to hack on GNUnet?

36 Want your PhD student to hack on GNUnet? A simple matter of installing the deps, right?

37 gnunet pkg-config-0.28 python glpk-4.56 gnurl gnutls libextractor-1.3 libgcrypt libidn-1.32 libmicrohttpd libltdl libunistring openssl-1.0.2d opus-1.1 pulseaudio-6.0 sqlite zlib bzip gdbm-1.11 libffi-3.1 readline-6.3 tcl tk gmp-6.0.0a perl groff texinfo-6.0 which-2.21 guile libtasn1-4.5 nettle libtiff libjpeg-9a exiv flac ffmpeg-2.8 file-5.22 glib gstreamer gst-plugins-base libogg libvorbis libgpg-error-1.19 curl m speex-1.2rc1 dbus libcap-2.24 eudev alsa-lib json-c-0.12 libsndfile libsamplerate intltool fftwf avahi check ncurses-6.0 libxft fontconfig libx libxext xproto libxrender freetype-2.6 gs-fonts-8.11 expat inputproto xextproto xtrans kbproto libxcb-1.11 libxau util-macros renderproto python-wrapper xcb-proto-1.11 libxslt libpthread-stubs-0.3 libxdmcp python-minimal libxml bison psutils-17 ghostscript netpbm flex procps xz python-wrapper lcms-2.6 libjpeg-8d libpng libpaper bison indent python bash libgc libatomic-ops bc-1.06 yasm ladspa-1.13 lame libass libbluray libcaca-0.99.beta19 libcdio-paranoia libquvi libtheora libvpx libx openal soxr twolame xvid gettext coreutils-8.24 tzdata-2015c gobject-introspection pango libxv cdparanoia-10.2 orc xmlto fribidi harfbuzz enca-1.16 doxygen texlive-2015 freeglut ftgl rc5 mesa libcdio-0.93 cyrus-sasl libquvi-scripts lua util-linux-2.27 net-base-5.3 tar-1.28 cairo graphite icu4c-55.1 recode libspectre poppler pixman python2-fonttools-2.5 acl attr openjpeg cairo python2-setuptools graphviz texlive-bin-2015 texlive-texmf-2015 gts gd libxaw libxmu libxpm libxt libice libsm fontforge b mpfr potrace-1.11 ruby tcsh teckit zziplib lua giflib libxi libspiro libuninameslist autoconf-2.69 automake-1.15 libtool zip-3.0 libxfixes fixesproto-5.0 autoconf-wrapper-2.69 autoconf-wrapper-2.69 libxrandr libxxf86vm xinput glu dri2proto-2.8 dri3proto-1.0 presentproto-1.0 libva-without-mesa libxvmc makedepend s2tc-1.0 glproto libdrm libxdamage libxshmfence-1.1 randrproto xf86vidmodeproto libxinerama gperf kmod-17 mesa-headers libpciaccess damageproto videoproto xineramaproto help2man libcddb gss libssh openldap mit-krb shishi bdb linux-pam libgcrypt fftw perl-xml-parser-2.44 libdaemon-0.14

38 $ guix environment --container gnunet... $ guix environment --ad-hoc python-ipython python-numpy \ -E ipython...

39

40 Whole-system deployment.

41 Linux-libre

42 Linux-libre initial RAM disk

43 Linux-libre initial RAM disk Guile

44 Linux-libre initial RAM disk Guile PID 1: GNU dmd services...

45 Linux-libre initial RAM disk Guile PID 1: GNU dmd services... Guile

46 Linux-libre initial RAM disk Guile PID 1: GNU dmd services... Guile applications

47 Trustworthiness.

48 Debian s dirtiest secret: Binary packages built by developers are used in the archive Lucas Nussbaum, FOSDEM 2015

49 Transparent binary/source deployment guix package --install=emacs The following package will be installed: emacs-24.5 /gnu/store/...-emacs-24.5 The following files will be downloaded: /gnu/store/...-emacs-24.5 /gnu/store/...-libxpm /gnu/store/...-libxext /gnu/store/...-libxaw

50 Transparent binary/source deployment guix package --install=emacs The following package will be installed: emacs-24.5 /gnu/store/...-emacs-24.5 The following files will be downloaded: /gnu/store/...-libxext /gnu/store/...-libxaw The following derivations will be built: /gnu/store/...-emacs-24.5.drv /gnu/store/...-libxpm drv

51 (define foo (package...)) user

52 (define foo (package...)) user test guix build foo /gnu/store/...-foo-1.0

53 (define foo (package...)) user test guix build foo /gnu/store/...-foo-1.0 git push git.sv.gnu.org

54 (define foo (package...)) user test guix build foo /gnu/store/...-foo-1.0 hydra.gnu.org build farm pull pull git push git.sv.gnu.org

55 (define foo (package...)) user test guix build foo /gnu/store/...-foo-1.0 get binary hydra.gnu.org build farm pull pull git push git.sv.gnu.org

56 (define foo (package...)) user test guix build foo /gnu/store/...-foo-1.0 pull git push git.sv.gnu.org

57 (define foo (package...)) user test guix build foo /gnu/store/...-foo-1.0 no maintainer uploads no single point of trust git push git.sv.gnu.org

58 source package recipes? binary hydra.gnu.org (define emacs (package... )) /gnu/store/... -emacs-24.5

59 The path to greater user control 1. Bit-reproducible builds 2. No single binary provider 3. Tools for users to challenge binaries

60 The path to greater user control 1. Bit-reproducible builds we have isolated build environments!... but we need builds to be deterministic 2. No single binary provider 3. Tools for users to challenge binaries

61 The path to greater user control 1. Bit-reproducible builds we have isolated build environments!... but we need builds to be deterministic 2. No single binary provider guix publish publish over GNUnet? (GSoC 2015) 3. Tools for users to challenge binaries

62 The path to greater user control 1. Bit-reproducible builds we have isolated build environments!... but we need builds to be deterministic 2. No single binary provider guix publish publish over GNUnet? (GSoC 2015) 3. Tools for users to challenge binaries

63 $ guix challenge --substitute-urls=" ht /gnu/store/...-openssl-1.0.2d contents differ: local hash: 0725l l zy4fma... /gnu/store/...-git contents differ: local hash: 00p3bmr nb mdqa9w... /gnu/store/...-pius contents differ: local hash: 0k4v3m k4v3m cy25x1...

64 Status.

65 Timeline Nov dubbed GNU Jan Apr , signed binaries, guix system July , installable operating system Jan , ARMv7 port... Aug Reproducibility in Parallel Computing Workshop (RepPar) 5 Nov , new service framework, etc.

66

67 Status full-featured package manager 2,600+ packages, 4 platforms Guix System Distribution β binaries at tooling: auto-update, linting, etc. l10n: 8 languages!

68 25 contributors each month... and lots of friendly people! 400 commits per month new packages per release

69 your help needed! install the distribution use it, report bugs, add packages help with the infrastructure + admin donate hardware/money share your ideas!

70

71 Copyright c 2010, 2012, 2013, 2014, 2015 Ludovic Courtès ludo@gnu.org. GNU Guix logo, GFDL, Copyright of other images included in this document is held by their respective owners. This work is licensed under the Creative Commons Attribution-Share Alike 3.0 License. To view a copy of this license, visit or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. At your option, you may instead copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is available at The source of this document is available from