Software Deployment Packaging, Installation, Updates

Transcription

1 Software Deployment Packaging, Installation, Updates Peter Libič, Pavel Parízek, Tomáš Kalibera DISTRIBUTED SYSTEMS RESEARCH GROUP CHARLES UNIVERSITY PRAGUE Faculty of Mathematics and Physics

2 What is software deployment Process of making a software system available to the user Packaging Distribution Installation Updates Uninstall

3 Key challenges

4 Key challenges Respecting standards and conventions of a target platform Dependency hell Configuration and user files Automation Robustness

5 Respecting standards and conventions File system hierarchy Putting binaries and configuration files at proper place Unix/Linux: /usr/bin, /opt, /etc, Windows: C:\Program Files, Names of packages and files Examples linux tar.gz kernel fc9.i586.rpm Use of the same mean of deployment as Target system/platform Required third-party libraries Easier resolving of dependencies

6 File system hierarchy: Unix Binaries: /bin, /sbin, /usr/bin Libraries: /lib, /usr/lib, /var/lib Configuration: /etc (general), /home/xxx (user specific) Prefixes /var (variable content, RW) /usr (secondary hierarchy for apps, RO) /opt/xxx (zero installed packages) /home (user directories) Filesystem Hierarchy Standard (FHS) - Linux Very detailed: core utilities, applications Additional conventions by distributions

7 File system hierarchy: Windows Binaries and program data: C:\Program Files\XXX (+special semantics of some subdirs by MS) Libraries: C:\Windows\System32 Configuration (user specific & general) C:\Documents and Settings\XXX Windows registry Database of configuration and data

8 File system hierarchy: Mac OS/X Applications /Applications/XXX (complete application distribution in a single archive - image file.dmg) Core system /System /Library configuration Home directories /Users/XXX

9 Dependency hell: how to install, upgrade with Conflicting dependencies Different software systems may depend on different versions of shared libraries, which may not necessarily be able to co-exist Many dependencies, long dependency chains, circular dependencies Hard to resolve Software requires a small part of a big library/many libraries All have to be downloaded, installed, including dependencies All have to be ported, kept up to date

10 Fighting the dependency hell Software packages with reasonable metadata Versions i.e. major and minor Dependencies described at good granularity Manual Tweaking of source code (works for conflicts) Tracking and installation of all dependencies by hand Automated Static linking of dependencies ( huge binary), independent software Repository-based package management systems (a single repository for an OS) The problem is solved by distribution maintainers

11 Profound instances of dependency hell JARs All linking is dynamic in Java Extremely difficult to fight by end users Hierarchy of class loaders Applications run in a single process (JVM), which has a (single) classpath RPMs Packages from different distributions with conflicting dependencies, package names

12 Profound instances of dependency hell (mostly resolved by now): DLLs Over-use of shared libraries Overwriting libraries by applications, no versioning Central COM registry NOW processes can have their own (versions) of libraries Dependencies can be specified Registration-free COM

13 Configuration and user files Update Issue: what to do with configuration files User may have modified the original configuration Solutions Preserve user s configuration and save the new configuration under a different file name User has to update the configuration manually (e.g. to a new file format) Uninstall Issue: what to do with configuration and user files User files: documents, saved files from games, Solutions Preserve them (for re-install in future), backup, remove Usually it is up to the user to decide

14 Other challenges of software deployment Automation One-step installation process Using a single command Automated updates downloading, installation Robustness Target system has to be left in a consistent state No partial installation Use of transactions

15 Means of software deployment

16 Means of software deployment Source code Zero installation Package management system Application-specific installer

17 Source code Advantage Allows installation of selected features only and high degree of optimization Depending e.g. on the configuration of target system, user preferences, available libraries and CPU architecture Drawback User has to perform some operations manually download, configuration, compilation Typical for GNU software./configure && make && make install (Autotools)

18 Source code distribution of updates Two options of distribution Patch Based on diff between old and new version Has to be applied manually to the old source code Archive with new version Directory with unpacked old source code is removed Update process 1.Uninstall of old version (via make uninstall, or rm rf /opt/xxx for self-contained installations) 2. Application of patch or unpacking of archive 3. Compilation and installation of a new version make; make install

19 Zero installation No explicit installation Copying of files to target location Manual post-installation configuration Setting some paths, Typical for Java programs Distributed as a set of binary files (JAR, class) and configuration files (xml, properties) in an archive Archive can be created by Ant or Maven Ant tasks: jar, zip

20 Package management system Collection of tools for automated and consistent processing of software packages Key concepts Package Means of distribution of software and metadata Metadata Version, description, list of dependencies, Local package database Information about installed packages (metadata) Software repository Remote storage of all available packages

21 Package management system tasks Downloading of a package Identification of dependencies (required packages) Installation Including all required dependencies Updating Including updates to (some) dependencies Uninstall Including some dependencies (not used by other programs) Querying of information about packages Installed on a system, available in a repository Getting sources for a package For an installed binary package, find and download the (exact) sources it was built from Building a package For a (specially structured and annotated) tree, build a package

22 Package management system examples System-level package managers Used in major Linux distributions RPM (Fedora, SUSE) DEB (Debian, Ubuntu) Ports (Gentoo Portage, FreeBSD, MacOS X - macports ) Windows installer (MSI files) App stores Android Market Apple App Store Application-level package managers CPAN (Perl Comprehensive Perl Archive Network) XPI (Firefox, Thunderbird) Eclipse plugins R project (CRAN)

23 Example System: APT

24 Apt package manager Installation apt-get install firefox -s option only print what would be done Removal apt-get remove firefox Upgrade apt-get upgrade, apt-get dist-upgrade Update of local cache of meta-data apt-get update

25 Apt package manager: source packages Getting sources of a package apt-get source firefox Downloads Original package code (tar.gz archive) Patch for distribution (added meta-data, bugfixes) Build meta-data Building retrieved sources Automatic on download or by dpkg-buildpackage Dependencies for building Installed automatically: apt-get build-dep firefox

26 Apt package manager: Queries Which package does an installed file come from? dpkg -S stdio.h (full path or name only) Which packages are installed? dpkg -l Fulltext search of descriptions of all available (not only installed) packages apt-cache search browser Which package (even uninstalled) would contain a given file? apt-file search stdio.h File lists are cached locally if apt-file is used

27 Apt package manager: Distributions Debian GNU/Linux (DEB packages) Ubuntu Conectiva (RPM packages) Redhat Fedora Suse And many others

28 Back to package managers overview

29 Package managers distribution of updates Two options Package with a complete new version Old version is removed/uninstalled Diff package Not so common, but still supported by some (delta RPM Suse, opensuse, Fedora) The issue is an efficient diff algorithm for compressed binary archives

30 Application-specific installer Installation program is a part of the distributed package Typical for Older Windows applications InstallShield Large proprietary applications Often do not respect standards and conventions Oracle Database, Borland Together, Windows today Installers usually based on Windows Installer framework by Microsoft Support in Visual Studio..

31 Package manager vs. installer Package management system Part of the operating system Software packages are not distributed with an installer Single database for all packages Single package format Application-specific installer Each application is distributed with its own installer No single database for all installed software Some information are stored in registry Multiple package formats

32 RedHat Package Manager/Packages (RPM)

33 RedHat Package Manager (RPM) Supports Installation rpm -ivh [package file] Example: rpm ivh foobar i386.rpm Uninstall rpm -e [package name] Example: rpm e foobar Updating rpm -Uvh [package file] Querying rpm -q [package name] Used in several Linux distributions Fedora, RHEL, SUSE Linux, CentOS, Mandriva, Part of Linux Standard Base (LSB)

34 RPM package contents Signature For verification of integrity Header (meta-data) Version Description List of dependencies Payload Archive with files

35 How to create a RPM package 1) Prepare the archive with source code Example: foo-1.2.tar.gz 2) Write a spec file Input for the rpm tool Tells RPM how to compile the software and create the package Example: foo.spec 3) Log in as root 4) Copy the archive with source code to the directory /usr/src/redhat/sources RPM assumes that this directory is used 5) Run the rpmbuild -ba [spec file] command Example: rpmbuild -ba foo.spec Creates two RPM packages /usr/src/redhat/srpms/foo src.rpm /usr/src/redhat/rpms/i386/foo i386.rpm

36 Example of spec file Summary: Foo Name: foo Version: 1.2 Release: 1 Source0: %{name}-%{version}.tar.gz License: GPL Group: Misc %description The Foo program does nothing interesting %prep %setup -q %build./configure make %install make install %files %defattr(-,root,root) /usr/local/bin/foo %doc COPYING README %config /etc/foo in /usr/src/redhat/sources header unpacks the source archive all files are installed as owned by root List of files to be included in the RPM package files tagged as documentation files tagged as configuration

37 Creation of RPM packages issues Software is installed on the build system during preparation of the RPM package Solution: build root Developer has to be logged in as root Solution: use different set of directories than /usr/src/redhat

38 Build root Idea Dummy directory tree is used for all installed files Example Group: Misc BuildRoot: %{_builddir}/%name-root %description %install rm -rf $RPM_BUILD_ROOT make DESTDIR=$RPM_BUILD_ROOT install %clean rm -rf $RPM_BUILD_ROOT %files /usr/src/redhat/build called at the end

39 Different set of directories Idea: change the %_topdir setting Create a file.rpmmacros in $HOME and add e.g. the following line to it: %_topdir /home/username/rpm The /home/username/rpm directory is used instead of /usr/src/redhat Soure code archive has to be copied into the /home/username/rpm/sources directory

40 Creation of RPM packages advanced topics Dependencies Virtual capabilities Installation scripts

41 Dependencies among RPM packages Required package Add requires: [package name] into the spec file Example requires: bar Required package of a specific version only Use requires: [package name] = [version number] Example requires: bar = Required package of a specific version or higher Use requires: [package name] >= [least version] Example requires: bar >= 2.1

42 Virtual capabilities Basic idea Same functionality can be provided by one from a set of packages Arbitrary name of a capability can be used in the specification of provision The provides clause Default value is the name of a package Example Tomcat package requires: java Packages for J2SE: Sun Java, GCJ + Classpath, provides: java

43 Installation scripts Motivation Non-trivial operations have to be performed before/after installation Update of runlevel information Automated configuration Special sections in spec file Pre-installation: %pre section Post-installation: %post section Each section contains a shell (bash) script Example %post /sbin/adduser apache # modify firewall configuration /sbin/service iptables restart

44 Open Build Service Open source web service Easy and automated rpm/deb package building

45 Ports Package manager family Based on (automated) source code building Originally in BSD systems Directories with makefiles and patches

46 Portage Package management system for Gentoo Key characteristics Building of packages from source code Optimization to HW and available libraries Packages defined via shell scripts (ebuilds) Similar to RPM spec files Organization of packages into categories e.g. app-editors, dev-perl,

47 Ebuild Shell script (bash) Description of the package Functions and instructions for download, unpacking, compilation, installation, Created for each version of the package Naming: pkg-ver[_suf][-r#].ebuild predefined suffixes: alpha, beta, pred2, rc3 Gentoo-specific revision number -r1, -r2, Example: eject r1.ebuild Stored in /usr/portage/category/pkgname

48 Ebuild example (part 1) # Copyright Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: $ inherit eutils header should not be modified DESCRIPTION="A command to eject a disc from the CD-ROM drive HOMEPAGE=" SRC_URI=" LICENSE="GPL-2 SLOT="0 KEYWORDS="~alpha amd64 arm hppa ia64 m68k mips ~ppc ~ppc64 s390 sh sparc x86 IUSE="nls DEPEND="!virtual/eject PROVIDE="virtual/eject S=${WORKDIR}/${PN} various shell functions...

49 Ebuild example (part 2)... header and definition of variables src_unpack() { unpack ${A} cd "${S} epatch "${FILESDIR}/${PN}" xmalloc.patch } src_compile() { econf $(use_enable nls) die emake die } invokes./configure uses automake src_install() { emake DESTDIR="${D}" install die "make install failed dodoc ChangeLog README PORTING TODO AUTHORS NEWS PROBLEMS }

50 Variables in ebuild Package name and version: P Package name: PN Source files: A Root build directory: WORKDIR Temporary build directory: S Temporary installation directory: D Image is then copied to the real installation directory List of use flags used in the ebuild: IUSE

51 Functions in ebuilds Control of building and installation process src_unpack unpack sources, run autotools src_compile configure and build package src_install Install into the directory specified by ${D} variable pkg_config... Initial configuration To be written by the developer

52 Functions in ebuilds cont. Helper functions use feature returns true or false use_with feature returns --with-feature or --without-feature use_enable feature Returns --enable-feature or --disable-feature Provided by the ebuild system

53 USE flags Mechanism for definition of optional features that should be included in (or excluded from) packages during building e.g. support for DVD, mp3, Reflected in compile-time options by ebuilds USE flag feature is translated to./configure --with-feature

54 Specification of dependencies via DEPEND variable Dependencies on specific package versions are supported Example DEPEND= virtual/kde X11-libs/gtk+-2 category has to be always included >=sys-libs/glibc-2.2.5

55 Emerge tool Command-line utility Primary user interface for Portage What it does Executes ebuilds for individual packages building, installation, Maintains database of installed packages Automated resolving of dependencies

56 Building from sources vs. binary packages Dependencies Sources: flexibility via compile-time options Optimization for libraries available on target system Binaries: several versions for typical cases Exponential number of possible configurations Installation time Sources: compilation is (very) slow Precompiled binary packages available for large programs LibreOffice, Firefox, Binaries: customization often not possible

57 Links Other (Windows directory layout) Package management cheat-sheet RPM Portage