The Hydra/Nix Approach to Continuous Integration

Transcription

1 1 / 21 The Hydra/Nix Approach to Continuous Integration Ludovic Courtès INRIA SED Cépage Team-Project devexp.bor@inria.fr 5 November 2009

2 2 / 21 Outline 1 Build & Deployment with Nix 2 Continuous Integration with Hydra 3 The End

3 3 / 21 A Declarative Approach Derivations (aka. Build Jobs ) a system type a list of build inputs a build process an output

4 3 / 21 A Declarative Approach Derivations (aka. Build Jobs ) a system type "i686-linux" a list of build inputs other derivations a build process an arbitrary script an output file tree

5 4 / 21 The Nix Build Expression Language The derivation Primitive derivation { name = "foo"; system = "x86 64-linux"; builder = builtins.tofile "builder.sh" mkdir -p "$out" echo "Hello, world!" > "$out/some-result" ; }

6 5 / 21 Job Composition } derivation { name = "foo"; system = "x86 64-linux"; builder = builtins.tofile "builder.sh" mkdir -p "$out" echo "Hello, world!" > "$out/some-result" ;

7 5 / 21 Job Composition let dep = derivation { name = "foo"; system = "x86 64-linux"; builder = builtins.tofile "builder.sh" mkdir -p "$out" echo "Hello, world!" > "$out/some-result" ; };

8 5 / 21 Job Composition let dep = derivation { name = "foo"; system = "x86 64-linux"; builder = builtins.tofile "builder.sh" mkdir -p "$out" echo "Hello, world!" > "$out/some-result" ; }; in derivation { name = "bar"; system = "x86 64-linux"; builder = builtins.tofile "builder.sh" mkdir -p "$out" ln -s "${dep}/some-result" "$out/my-result" ; }

9 6 / 21 Complex Nix Expressions ( Job Procedures ) GNU Hello {fetchurl, stdenv}: stdenv.mkderivation { name = "hello-2.3"; src = fetchurl { url = mirror://gnu/hello/hello-2.3.tar.bz2; sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1"; }; } meta = { description = "A program that produces a friendly greeting"; homepage = };

10 6 / 21 Complex Nix Expressions ( Job Procedures ) GNU Hello {fetchurl, stdenv, gettext}: stdenv.mkderivation { name = "hello-2.3"; src = fetchurl { url = mirror://gnu/hello/hello-2.3.tar.bz2; sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1"; }; buildinputs = [ gettext ]; meta = { description = "A program that produces a friendly greeting"; homepage = }; }

11 7 / 21 Dependency Graph for GNU Hello Run-Time Dependencies linux-headers glibc-2.9 hello-2.3

12 default-builder.sh ncurses-5.7.drv lzma drv texinfo-4.13a.drv perl drv linux-headers drv binutils drv replace-2.24.drv pcre-7.8.drv gawk drv lzma drv gettext-0.17.drv perl drv gnused drv bzip drv gnum drv gnupatch drv patchelf-0.4.drv gnumake-3.81.drv gnugrep drv prehook.sh bootstrap-tools.drv builder.sh setup.sh libtool-2.2.6a.drv attr drv acl drv no-sys-dirs.patch builder.sh ld-wrapper.sh bootstrap-glibc.drv utils.sh setup-hook.sh add-flags gcc-wrapper.sh setup-hook.sh malloc.patch gettext-fix.patch stdenv-linux-boot.drv builder.sh log.patch impure-dirs.patch diffutils drv coreutils-7.2.drv bootstrap-gcc.drv prehook.sh builder.sh stdenv-linux-boot.drv nss-skip-unavail.patch glibc-2.9.drv locale-override.patch rpcgen-path.patch gcc-wrapper drv new-dtags.patch bootstrap-gcc.drv stdenv-linux-boot.drv pass-cxxcpp.patch builder.sh no-sys-dirs.patch linux tar.bz2.drv gnum drv gcc drv gmp drv mpfr drv glibc tar.bz2.drv binutils tar.bz2.drv mpfr tar.bz2.drv gmp tar.bz2.drv texinfo-4.13a.tar.lzma.drv ncurses-5.7.tar.gz.drv perl tar.gz.drv gcc-core tar.bz2.drv m tar.bz2.drv lzma tar.gz.drv sed tar.gz.drv gcc-g tar.bz2.drv bash drv attr_ tar.gz.drv bzip tar.gz.drv bash drv bash drv coreutils-7.2.tar.gz.drv bash drv bison-2.3.tar.bz2.drv bash drv bash-4.0.tar.gz.drv bash drv bash drv bash drv gettext-0.17.tar.gz.drv libtool-2.2.6a.tar.lzma.drv bash drv bash drv bash drv bash drv replace-2.24-src tar.gz.drv bash drv bash drv bash drv bash drv bash drv diffutils tar.gz.drv gawk tar.bz2.drv acl_ tar.gz.drv pcre-7.8.tar.bz2.drv grep tar.bz2.drv patch tar.gz.drv findutils tar.gz.drv patchelf-0.4.tar.bz2.drv make-3.81.tar.bz2.drv gzip tar.gz.drv bison-2.3.drv write-mirror-list.sh gcc-wrapper drv findutils-path.patch change_echo_path.patch findutils drv mirrors-list.drv perl drv stdenv-linux.drv hello-2.3.drv bash-4.0-p17.drv hello-2.3.tar.bz2.drv bzip2 mkdir curl.bz2 download.sh ln cpio sh openssl-0.9.8k.drv curl drv bootstrap-tools.cpio.bz2.drv connect-timeout.patch zlib drv unpack-bootstrap-tools.sh gnulib-futimens.patch gzip drv gnutar-1.22.drv implausible.patch mirrors-list.drv stdenv-linux-boot.drv builder.sh tar-1.22.tar.bz2.drv zlib tar.gz.drv openssl-0.9.8k.tar.gz.drv curl tar.bz2.drv 7 / 21 Dependency Graph for GNU Hello Compile-Time Dependencies 147 nodes 535 edges

13 bzip gdbm python eject linux-pam cracklib libvolume_id libxslt udev-125 neon libxml pciutils hal bash-4.0-p17 curl libusb usbutils-0.73 libwpd linux-headers zlib ncurses-5.7 gamin gnome-vfs libgsf iniparser-3.0b cups openssl-0.9.8k readline-6.0 samba heimdal libpng db cyrus-sasl gmp openldap mpfr gcc libtiff libunwind libjpeg-6b glibc-2.9 libidl popt-1.15 ORBit libbonobo glib perl dbus-glib-0.80 dbus-library openoffice.org expat jasper atk attr acl GConf coreutils-7.2 freetype fontconfig icu4c-3.6 pixman gtk gnused binutils cairo pango libice e2fsprogs libsm xlibs-wrapper libxft renderproto libxrender xproto xcb-util kbproto libxcb-1.2 libxinerama libxdmcp xineramaproto libx randrproto libxrandr xextproto libxau libxext printproto libxt libxp libxtst libxi jdk-1.6.0_13 recordproto inputproto / 21 Dependency Graph for OpenOffice.org Run-Time Dependencies 94 nodes 426 edges

14 setup.sh freetype drv builder.sh gperf drv openssl-0.9.8k.drv classr.patch register-race-fix.patch cygwin-4.5.patch builder.sh db drv builder.sh perl drv perl-compress-raw-zlib drv libusb drv zlib drv e2fsprogs drv setup-hook.sh search-path.patch python drv boost drv bzip drv builder.sh perl-io-compress-base drv perl-io-compress-zlib drv perl-compress-zlib drv no-usr.patch install.patch builder.sh setup-hook.sh requires-private.patch Compress-Raw-Zlib tar.gz.drv IO-Compress-Base tar.gz.drv Compress-Zlib tar.gz.drv IO-Compress-Zlib tar.gz.drv libsndfile tar.gz.drv pkg-config-0.23.tar.gz.drv boost_1_38_0.tar.bz2.drv zip232.tgz.drv file-5.03.tar.gz.drv libmspack alpha.tar.gz.drv libwpd tar.gz.drv libgsf tar.bz2.drv samba tar.gz.drv iniparser3.0b.tar.gz.drv groff tar.gz.drv openldap tgz.drv cyrus-sasl tar.gz.drv db tar.gz.drv heimdal tar.gz.drv readline-6.0.tar.gz.drv libunwind tar.gz.drv usbutils-0.73.tar.gz.drv usb.ids bz2.drv hal tar.gz.drv libsmbios tar.gz.drv libusb tar.gz.drv gamin tar.gz.drv Python tar.bz2.drv gdbm tar.gz.drv libvolume_id tar.bz2.drv udev-125.tar.bz2.drv pci.ids bz2.drv pciutils tar.bz2.drv eject tar.gz.drv GConf tar.bz2.drv gettext-0.17.tar.gz.drv dbus-glib-0.80.tar.gz.drv expat tar.gz.drv dbus tar.gz.drv setup-hook.sh glib tar.bz2.drv XML-Parser-2.36.tar.gz.drv gnome-mime-data tar.bz2.drv libidl tar.bz2.drv ORBit tar.bz2.drv popt-1.15.tar.gz.drv libxml2-sources tar.gz.drv libbonobo tar.bz2.drv intltool tar.bz2.drv gnome-vfs tar.bz2.drv bison-2.3.tar.bz2.drv atk tar.bz2.drv jasper zip.drv ncurses-5.7.tar.gz.drv pango tar.bz2.drv cairo tar.gz.drv flex-2.5.4a.tar.gz.drv xcb-proto-1.4.tar.bz2.drv libpthread-stubs-0.1.tar.bz2.drv libxcb-1.2.tar.bz2.drv m tar.bz2.drv xcb-util tar.bz2.drv gperf tar.gz.drv icu-3.8-setbreaktype-public.diff?rev=1.1.drv icu4c-3_6-src.tgz.drv pixman tar.bz2.drv libxft tar.bz2.drv freetype tar.bz2.drv fontconfig tar.gz.drv libxrender tar.bz2.drv renderproto tar.bz2.drv xineramaproto tar.bz2.drv libxinerama tar.bz2.drv libxslt tar.gz.drv libpng tar.bz2.drv libice tar.bz2.drv gtk tar.bz2.drv libxrandr tar.bz2.drv randrproto tar.bz2.drv libxdmcp tar.bz2.drv Archive-Zip-1.16.tar.gz.drv libsm tar.bz2.drv e2fsprogs tar.gz.drv xtrans tar.bz2.drv builder.sh getopt tar.gz.drv tcsh tar.gz.drv which-2.20.tar.gz.drv neon tar.gz.drv patch-ltmain.sh.drv patch-ltconfig.drv jpegsrc.v6b.tar.gz.drv libtool tar.gz.drv tiff tar.gz.drv Sablot tar.gz.drv xproto tar.bz2.drv libxmu tar.bz2.drv xextproto tar.bz2.drv libxext tar.bz2.drv libxau tar.bz2.drv bigreqsproto tar.bz2.drv xf86bigfontproto tar.bz2.drv xcmiscproto tar.bz2.drv libx tar.bz2.drv kbproto tar.bz2.drv libxaw tar.bz2.drv libxt tar.bz2.drv printproto tar.bz2.drv libxpm tar.bz2.drv inputproto tar.bz2.drv libxi tar.bz2.drv libxp tar.bz2.drv recordproto tar.bz2.drv libxtst tar.bz2.drv unzip552.tar.gz.drv jdk-6u13-dlj-linux-amd64.bin.drv OOo_3.0.1_src_system.tar.bz2.drv OOo_3.0.1_src_core.tar.bz2.drv cracklib tar.gz.drv cracklib-words.gz.drv builder.sh Linux-PAM tar.bz2.drv cups source.tar.bz2.drv apache-ant bin.tar.bz2.drv ant-contrib-1.0b3-bin.tar.bz2.drv iniparser-3.0b.drv groff drv openldap drv cyrus-sasl drv heimdal drv default-builder.sh gdbm drv libvolume_id drv udev-125.drv pciutils drv gettext-0.17.drv samba drv hal drv prehook.sh perl-xml-parser-2.36.drv expat drv add-flags eject drv gnome-mime-data drv ld-wrapper.sh libsndfile drv utils.sh gettext-0.17.drv zip-2.32.drv file-5.03.drv libmspack alpha.drv gamin drv gnome-vfs drv libgsf drv attr drv glib drv dbus-glib-0.80.drv acl drv perl drv libtool-2.2.6a.drv lzma drv libunwind drv usbutils-0.73.drv perl drv libsmbios drv intltool drv GConf drv dbus-library drv libbonobo drv neon drv libxml drv popt-1.15.drv bootstrap-glibc.drv stdenv-linux-boot.drv openssl-0.9.8k.tar.gz.drv bzip tar.gz.drv attr_ tar.gz.drv gettext-0.17.tar.gz.drv perl tar.gz.drv no-sys-dirs.patch setup-hook.sh libtool-2.2.6a.tar.lzma.drv acl_ tar.gz.drv zlib tar.gz.drv make-3.81.tar.bz2.drv diffutils tar.gz.drv lzma tar.gz.drv m tar.bz2.drv gzip tar.gz.drv bash drv gmp tar.bz2.drv binutils tar.bz2.drv glibc tar.bz2.drv linux tar.bz2.drv mpfr tar.bz2.drv gcc-core tar.bz2.drv gcc-g tar.bz2.drv texinfo-4.13a.tar.lzma.drv ncurses-5.7.tar.gz.drv bash-4.0.tar.gz.drv bison-2.3.tar.bz2.drv bash drv replace-2.24-src tar.gz.drv tar-1.22.tar.bz2.drv bash drv bash drv patch tar.gz.drv sed tar.gz.drv gawk tar.bz2.drv grep tar.bz2.drv pcre-7.8.tar.bz2.drv bash drv bash drv patchelf-0.4.tar.bz2.drv bash drv bash drv bash drv bash drv bash drv bash drv bash drv bash drv bash drv bash drv bash drv coreutils-7.2.tar.gz.drv findutils tar.gz.drv curl tar.bz2.drv no-kerberos.patch stdenv-linux-boot.drv stdenv-linux-boot.drv bootstrap-gcc.drv gcc-wrapper drv atk drv ORBit drv gnum drv readline-6.0.drv gcc-wrapper drv bootstrap-gcc.drv setup-hook.sh builder.sh ncurses-5.7.drv gcc-wrapper.sh libwpd drv fontconfig drv libidl drv bison-2.3.drv flex-2.5.4a.drv linux-headers drv jasper drv stdenv-linux-boot.drv diffutils drv gnum drv lzma drv pango drv gnulib-futimens.patch gzip drv gmp drv gtk drv coreutils-7.2.drv new-dtags.patch binutils drv gnumake-3.81.drv gnupatch drv stdenv-linux.drv nss-skip-unavail.patch builder.sh locale-override.patch rpcgen-path.patch glibc-2.9.drv mpfr drv builder.sh gcc drv no-sys-dirs.patch pass-cxxcpp.patch ncurses-5.7.drv texinfo-4.13a.drv bison-2.3.drv prehook.sh cairo drv gawk drv pcre-7.8.drv gnugrep drv mirrors-list.drv impure-dirs.patch log.patch malloc.patch implausible.patch gettext-fix.patch patchelf-0.4.drv write-mirror-list.sh pkg-config-0.23.drv cpio bzip2 sh ln bootstrap-tools.drv download.sh bootstrap-tools.cpio.bz2.drv curl.bz2 mkdir unpack-bootstrap-tools.sh gnum drv pixman drv libpng drv openoffice.org drv perl-archive-zip-1.16.drv libxslt drv tcsh drv xcb-util drv xcb-proto-1.4.drv libtool drv which-2.20.drv sablotron drv libxcb-1.2.drv libxft drv xlibs-wrapper.drv libpthread-stubs-0.1.drv icu4c-3.6.drv replace-2.24.drv gnutar-1.22.drv gnused drv builder.sh bash-4.0-p17.drv setup-new.sh connect-timeout.patch stdenv-linux.drv curl drv findutils-path.patch change_echo_path.patch mirrors-list.drv libxrender drv renderproto drv xineramaproto drv libxinerama drv randrproto drv libxdmcp drv libxrandr drv libice drv xtrans drv libsm drv findutils drv builder.sh xproto drv xextproto drv libxmu drv libxext drv libxaw drv libxau drv libxpm drv libx drv libxt drv bigreqsproto drv libxi drv xf86bigfontproto drv xcmiscproto drv kbproto drv inputproto drv libxp drv jdk-1.6.0_13.drv printproto drv libxtst drv recordproto drv jdk6-construct.sh getopt drv dlj-bundle-builder.sh builder.sh libtiff drv builder.sh libjpeg-6b.drv make-wrapper.sh hook.drv cups drv unzip-5.52.drv builder.sh ant-jdk-1.6.0_13.drv cracklib drv linux-pam drv ant-contrib-1.0b3.drv oo.patch builder.sh 8 / 21 Dependency Graph for OpenOffice.org Compile-Time Dependencies 411 nodes 2043 edges

15 9 / 21 Compilation Result Caching ( Memoization ) Store Paths $ nix-build -A hello

16 9 / 21 Compilation Result Caching ( Memoization ) Store Paths $ nix-build -A hello /nix/store/ksaxbhsnwmyxilx5ha8k704cp2iabh0y-hello-2.3

17 9 / 21 Compilation Result Caching ( Memoization ) Store Paths $ nix-build -A hello /nix/store/ksaxbhsnwmyxilx5ha8k704cp2iabh0y-hello-2.3 How Caching Works all inputs explicitly given to derivation output path = hash of all build inputs

18 10 / 21 Dependencies Among Derivation Outputs Problem: What Does a Build Output Depend On? a build result may depend on a previous build result example: a program linked with a library linux-headers glibc-2.9 hello-2.3

19 10 / 21 Dependencies Among Derivation Outputs Problem: What Does a Build Output Depend On? a build result may depend on a previous build result example: a program linked with a library linux-headers glibc-2.9 Solution: Conservatively Scan Output Files scan output files for occurrences of /nix/store/* maintain a dependency graph hello-2.3

20 11 / 21 Support for Distributed Builds The Build Hook derivation realization can be deferred to a hook the hook can accept, reject, or postpone

21 11 / 21 Support for Distributed Builds The Build Hook derivation realization can be deferred to a hook the hook can accept, reject, or postpone the build-remote hook: 1 finds a host for the system type (e.g., i686-linux) 2 copies missing inputs to the remote host 3 copies output(s) back to the build host

22 11 / 21 Support for Distributed Builds The Build Hook derivation realization can be deferred to a hook the hook can accept, reject, or postpone the build-remote hook: 1 finds a host for the system type (e.g., i686-linux) 2 copies missing inputs to the remote host 3 copies output(s) back to the build host the foundation for Hydra s distributed builds

23 12 / 21 Outline 1 Build & Deployment with Nix 2 Continuous Integration with Hydra 3 The End

24 Hydra: Distributed Continuous Integration with fancy web interface (not as fancy as Xooctory s) 13 / 21

25 Hydra: Distributed Continuous Integration with fancy web interface (not as fancy as Xooctory s) 13 / 21

26 Hydra: Distributed Continuous Integration with fancy web interface (not as fancy as Xooctory s) 13 / 21

27 Hydra: Distributed Continuous Integration with fancy web interface (not as fancy as Xooctory s) 13 / 21

28 14 / 21 Jargon Mapping Xooctory job plan job Hydra project, jobset, job build

29 14 / 21 Jargon Mapping Xooctory job plan job what (JobResource) when (JobTrigger) how (JobProcedure, ResultCollector) Hydra project, jobset, job build build inputs (Nix) SCM change (Hydra) builder (Nix), build products (Hydra)

30 15 / 21 Types of Build Results Results Written to $out/nix-support/hydra-build-products. build logs binaries: Nix store paths,.debs,.rpms source tarballs ISO 9660 CD images code coverage reports screenshots

31 Types of Build Results 15 / 21

32 Types of Build Results 15 / 21

33 16 / 21 Architecture Data Storage the Nix store, used as a build cache database for project info, available builds, etc. 3 Processes the scheduler the queue runner the web interface

34 16 / 21 Architecture Data Storage the Nix store, used as a build cache database for project info, available builds, etc. 3 Processes the scheduler : looks for SCM changes, queues builds the queue runner the web interface

35 16 / 21 Architecture Data Storage the Nix store, used as a build cache database for project info, available builds, etc. 3 Processes the scheduler : looks for SCM changes, queues builds the queue runner : executes queued builds the web interface

36 16 / 21 Architecture Data Storage the Nix store, used as a build cache database for project info, available builds, etc. 3 Processes the scheduler : looks for SCM changes, queues builds the queue runner : executes queued builds the web interface : click!

37 17 / 21 Job Scheduling 1 upon SCM change, enqueue builds for x86 64-linux, i686-freebsd, etc. 2 pick up a build from the queue 3 build it, relying on a build hook to distribute builds 4 the build hook copies missing input, then fetches the result

38 17 / 21 Job Scheduling 1 upon SCM change, enqueue builds for x86 64-linux, i686-freebsd, etc. 2 pick up a build from the queue 3 build it, relying on a build hook to distribute builds 4 the build hook copies missing input, then fetches the result Shortcomings little information available to the build hook e.g., accepts/rejects builds without knowing its inputs can t make decisions based on input availability

39 18 / 21 Outline 1 Build & Deployment with Nix 2 Continuous Integration with Hydra 3 The End

40 19 / 21 Summary Building with Nix complete dependency graph through Nix expressions automatically maintained dependency graph among outputs automatic build result cache

41 19 / 21 Summary Building with Nix complete dependency graph through Nix expressions automatically maintained dependency graph among outputs automatic build result cache Continuous Integration with Hydra fancy web interface (RESTful) distributed builds on heterogeneous machines currently simplistic build scheduling TU Delft: 2500 jobs, 36 cores, 6 system types (Nov. 2009)

42 Thanks! 20 / 21

43 21 / 21 Links Web Nix, NixOS, and Hydra Papers papers Hydra instance for NixOS, etc. E. Dolstra and A. Hemel, Purely Functional System Configuration Management, HotOS XI, May E. Dolstra and E. Visser, Hydra: A Declarative Approach to Continuous Integration, 2008, submitted. E. Dolstra, Nix User Guide,