ZeroVM Package Manager Documentation

Transcription

1 ZeroVM Package Manager Documentation Release ZeroVM Team October 14, 2014

2

3 Contents 1 Introduction Creating a ZeroVM Application ZeroCloud Authentication Config Auth v Auth v ZPM Commands zpm new zpm bundle zpm deploy zpm execute zpm help zpm version The zapp.yaml File The meta Section The execution Section The help Section The bundling Section The ui Section Packaging 15 6 Glossary 17 7 API ZPM Core Functions Indices and tables 25 Python Module Index 27 i

4 ii

5 Contents: Contents 1

6 2 Contents

7 CHAPTER 1 Introduction The ZeroVM Package Manager, ZPM, is the tool you use to create and deploy ZeroVM applications. 1.1 Creating a ZeroVM Application We will use a simple Python application as our running example. Note: As of version 0.1, ZPM only supports Python applications with no third-party dependencies. To get started, we will show how to package a very simple Hello World application. There will be just one file, hello.py, with the expected content: print "Hello from ZeroVM!" To bundle this into a zapp, ZPM needs a configuration file called zapp.yaml Configuration File The zapp.yaml file will look like this: meta: Version: "0.1" name: hello Author- Martin Geisler <martin@geisler.net> Summary: A small Hello World app execution: groups: - name: hello path: file://python2.7:python args: /hello.py devices: - name: python2.7 - name: stdout help: description: A small "Hello World" app. bundling: - hello.py 3

8 The file is in YAML format and describes the program to execute, some meta data about it, help about the program and its arguments (this program has none), and finally information about which files to include when bundling. The different sections are described in more detail in The zapp.yaml File Bundling Simply running zpm bundle will create the hello.zapp: $ zpm bundle adding /home/mg/src/hello/hello.py adding /home/mg/src/hello/zapp.yaml created hello.zapp You see the files added to the zapp here it s simply hello.py together with the zapp.yaml file containing the meta data. You can now publish hello.zapp on your webserver, send it to your friends, etc. They will be able to run it after they deploy it like we describe next Deployment To deploy hello.zapp, you need access to a ZeroCloud cluster (Swift running the ZeroVM middleware). For help on configuring zpm to access a ZeroCloud cluster, see ZeroCloud Authentication Config. We will deploy it to a test container under the folder hello: $ zpm deploy test/hello hello.zapp deploying hello.zapp found token: MIIGjwYJKoZIhvcNAQcC... found Swift: uploading 398 bytes to test/hello/hello.zapp updated test/hello/hello.zapp succesfully For testing, you can execute the job after it has been deployed: $ zpm deploy test/hello hello.zapp --execute deploying hello.zapp found token: MIIGjwYJKoZIhvcNAQcC... found Swift: uploading 398 bytes to test/hello/hello.zapp updated test/hello/hello.zapp succesfully job template: [{ exec : { args : /hello.py, path : u file://python2.7:python }, devices : [{ name : u python2.7 }, { name : u stdout }, { name : image, path : u swift://account/test/hello/hello.zapp }], name : u hello }] executing <Response [200]> Hello from ZeroVM! There currently is no support for executing the application later. Issue #37 deals with that. 4 Chapter 1. Introduction

9 CHAPTER 2 ZeroCloud Authentication Config ZeroCloud is middleware for OpenStack Swift which enables ZeroVM applications to run on stored objects. Deploying applications to ZeroCloud using zpm requires authentication. zpm currently supports both v1 and v2 authentication schemes through the use of command-line flags and environment variables. See below for instructions and best practices for handling both versions. 2.1 Auth v1 Authentication parameters can be specified directly on the command-line like so: $ zpm deploy --auth \ --user tenant1:user1 \ --key f0ec1500-de68-42bd-8350-f671b50a79bd \ test_container hello.zapp Note that with v1, the --user is a concatentation of the tenant name and the username, contrasted with v2 where they are specified separately. (See below.) You can also specify the authenication parameters by setting the following environment variables: $ export ST_AUTH= $ export ST_USER=tenant1:user1 $ export ST_KEY=f0ec1500-de68-42bd-8350-f671b50a79bd Which shortens the zpm command to this: $ zpm deploy test_container hello.zapp For convenience, you can put the above export statements into a text file (called zerocloud_v1, for example) and source it to automatically set up your environment: $ source zerocloud_v1 2.2 Auth v2 Authentication parameters can be specified directly on the command-line like so: 5

10 $ zpm deploy --os-auth-url \ --os-username user1 \ --os-tenant-name tenant1 --os-password secret \ test_container hello.zapp --os-auth-url should be the public endpoint defined for Keystone. You can also specify the authenication parameters by setting the following environment variables: $ export OS_AUTH_URL= $ export OS_USERNAME=user1 $ export OS_PASSWORD=secret $ export OS_TENANT_NAME=tenant1 Which shortens the zpm command to this: $ zpm deploy test_container hello.zapp For convenience, you can put the above export statements into a text file (called zerocloud_v2, for example) and source it to automatically set up your environment: $ source zerocloud_v2 6 Chapter 2. ZeroCloud Authentication Config

11 CHAPTER 3 ZPM Commands The zpm script has the following top-level commands: 3.1 zpm new Create a default ZeroVM application zapp.yaml specification in the target directory. If no directory is specified, zapp.yaml will be created in the current directory. usage: zpm new [-h] [--with-ui] [--log-level {debug,info,warning,error,critical}] [WORKING_DIR] Positional arguments: Options: dir Non-existent or empty directory (default:.) --with-ui=false, -u=false Include user interface template files --log-level=warning, -l=warning Defaults to warn (default: warning) Possible choices: debug, info, warning, error, critical 3.2 zpm bundle This command creates a Zapp using the instructions in zapp.yaml. The file is read from the project root. usage: zpm bundle [-h] [--log-level {debug,info,warning,error,critical}] Options: --log-level=warning, -l=warning Defaults to warn (default: warning) Possible choices: debug, info, warning, error, critical 3.3 zpm deploy This deploys a zapp onto Swift. The zapp can be one you have downloaded or produced yourself with zpm bundle. 7

12 You will need to know the Swift authentication URL, username, password, and tenant name. These can be supplied with command line flags (see below) or you can set the corresponding environment variables. The environment variables are the same as the ones used by the Swift command line tool, so if you re already using that to upload files to Swift, you will be ready to go. usage: zpm deploy [-h] [--execute] [--summary] [--force] [--auth-version {1.0,2.0}] [--auth AUTH] [--user USER] [--key KEY] [--os-auth-url OS_AUTH_URL] [--os-tenant-name OS_TENANT_NAME] [--os-username OS_USERNAME] [--os-password OS_PASSWORD] [--no-ui-auth] [--log-level {debug,info,warning,error,critical}] target zapp Positional arguments: Options: target zapp --execute=false Deployment target (Swift container name) A ZeroVM application Immediately execute the deployed Zapp (for testing) --summary=false, -s=false Show execution summary table (use with execute ) --force=false, -f=false Force deployment to a non-empty container --auth-version, -V Swift auth version (default: 1.0) --auth, -A --user, -U --key, -K --os-auth-url --os-tenant-name --os-username --os-password --no-ui-auth=false Possible choices: 1.0, 2.0 (Auth v1.0) URL for obtaining an auth token (default: $ST_AUTH) (Auth v1.0) User name for obtaining an auth token (default: $ST_USER) (Auth v1.0) Key for obtaining an auth token (default: $ST_KEY) (Auth v2.0) OpenStack auth URL (default: $OS_AUTH_URL) (Auth v2.0) OpenStack tenant (default: $OS_TENANT_NAME) (Auth v2.0) OpenStack username (default: $OS_USERNAME) (Auth v2.0) OpenStack password (default: $OS_PASSWORD) Do not generate any authentication code for the web UI --log-level=warning, -l=warning Defaults to warn (default: warning) Possible choices: debug, info, warning, error, critical 3.4 zpm execute usage: zpm execute [-h] [--log-level {debug,info,warning,error,critical}] [--container CONTAINER] [--summary] [--auth-version {1.0,2.0}] [--auth AUTH] [--user USER] [--key KEY] [--os-auth-url OS_AUTH_URL] [--os-tenant-name OS_TENANT_NAME] [--os-username OS_USERNAME] [--os-password OS_PASSWORD] zapp Positional arguments: 8 Chapter 3. ZPM Commands

13 Options: zapp Name of the zapp to execute --log-level=warning, -l=warning Defaults to warn (default: warning) Possible choices: debug, info, warning, error, critical --container Swift container name (containing the zapp) --summary=false, -s=false Show execution summary table --auth-version, -V Swift auth version (default: 1.0) Possible choices: 1.0, auth, -A (Auth v1.0) URL for obtaining an auth token (default: $ST_AUTH) --user, -U (Auth v1.0) User name for obtaining an auth token (default: $ST_USER) --key, -K (Auth v1.0) Key for obtaining an auth token (default: $ST_KEY) --os-auth-url (Auth v2.0) OpenStack auth URL (default: $OS_AUTH_URL) --os-tenant-name (Auth v2.0) OpenStack tenant (default: $OS_TENANT_NAME) --os-username (Auth v2.0) OpenStack username (default: $OS_USERNAME) --os-password (Auth v2.0) OpenStack password (default: $OS_PASSWORD) 3.5 zpm help usage: zpm help [-h] [command] Positional arguments: command A zpm command 3.6 zpm version usage: zpm version [-h] 3.5. zpm help 9

14 10 Chapter 3. ZPM Commands

15 CHAPTER 4 The zapp.yaml File The zapp.yaml plays a central role when writing an application for deployment on ZeroCloud. This is where you specify things such as: Meta-data about the zapp, such as the name of the author and the version number. Where to find the NaCl executables ( nexes ) and other files that make up the ZeroVM applicaiton. These are the files that zpm bundle include into the zapp. Arguments that must be specifed when the application is execution. These will be invocation specific. A sample zapp.yaml file for a simple Hello World application looks like this: meta: Version: "0.1" name: hello Author- Your Name <your.name@example.net> Summary: A small Hello World app execution: groups: - name: hello path: file://python2.7:python args: /hello.py devices: - name: python2.7 - name: stdout help: description: Enter your name and you will be greeted args: - [name, Your name] bundling: - hello.py The file is a simple YAML document. At the top, there is a mapping with a number of keys which we will call sections in the following. Each section describes part of the zapp file produced. We will go throught the sections now. 11

16 4.1 The meta Section This section describes the generated zapp. The meta data here is currently unused, but we expect it to be used for a future zapp repository, i.e., a website where you can upload a zapp and let others download it. The keys in this section are: name The short name of your zapp. summary A short summary of what your zapp does. author- Your name and . version The version number of your zapp. 4.2 The execution Section This section describes the runtime behavior of your zapp: which groups of nodes to create and which nexe to invoke for each. The groups key is a list of individual groups. Each group has these keys: name The name of this group. You use reference this name when you connect a group of nodes to another group. path Path to the nexe that all nodes in this group will execute. You will typically specify this as file://image:nexe, which means that the nexe is found in the system image called image under the name nexe. The path to the nexe is relative to the root of the system image. args Command line arguments that will be passed to the nexe. devices List of devices that this group need. Each device has a name which determines the type of device. These are the standard devices that are always present: connect stdin This device feeds standard input to your program. You need to specify a path as well as name. The path can be a swift:// URL pointing to an object, which will make ZeroCloud execute the application on a Swift nodes that holds his object. stdout This device captures the standard output of your program. If you don t specify a path, the output is simply passed back to you when you invoke the program. This is how the default web UI shows the program output. If you do specify a swift:// URL in path, the output is stored there. stderr This device captures the standard output of your program. You need to specify where the output should be stored using a swift:// URL in path. Otherwise the error output will be discarded. In addition a ZeroCloud installation can offer a number of system images. They will have to be installed by the system adminitrator of the system your users deploy the zapp onto. Referencing a system image will cause it to be mounted as the root filesystem when nexe is executed. These are the initially supported system images: python27 This gives you a Python 2.7 environment. The interpreter should be specified as file://python27:python in the path key. List of other groups that this group should be connected with. Before the execution starts, devices will automatically be created to connect the nodes in the groups. If a group with n nodes named foo connects to a group with m nodes called bar, then. Nodes in the foo group will find devices named: /dev/out/bar-1 /dev/out/bar-2... /dev/out/bar-m 12 Chapter 4. The zapp.yaml File

17 count corresponding to each of the m instances in the bar group. Each of the n nodes in bar will find these devices: /dev/in/foo-1 /dev/in/foo-2... /dev/in/foo-n If there is only a single node in a group, the corresponding device is named /dev/out/bar or /dev/in/foo. What is written on channel in /dev/out appears on the corresponding channel in /dev/in. Defaults to 1. This can be used to specify the number of nodes in a group that would otherwise just have a single node, i.e., because the node writes to a single output object. The count is ignored if a device path contains a wildcard. replicate attach Defaults to 1 which means no replication, other supported values are 2 and 3. This will make ZeroCloud run multiple copies of the nodes in the group. Override the strategy used by ZeroCloud to place nodes in the Swift cluster. By default, ZeroCloud will start jobs on the node holding the input and output objects. When attach specifies a device that has a swift:// path, ZeroCloud will run the job with on the node holding this object. 4.3 The help Section This section allows you to describe the command line arguments needed for your application. It is used when you let zpm auto-generate a web UI for your application. The keys are: description A short description, similar to what programs print when invoked with no arguments. args A list of arguments. Each list entry is a tuple (really a two-element list) with the name of the argument and a corresponding help text. 4.4 The bundling Section For zpm bundle to work, it needs to know which files to include in the zapp. You specify them here as a list of glob patterns (such as src/*.py). The patterns are expanded relative to the project root, i.e., the directory containing the zapp.yaml file. 4.5 The ui Section You can optionally include a ui section. If it is left out, zpm will create a simple web UI for you. The section works like the bundling section: you specify a list of glob patterns and these files will be included in the zapp. The UI files are extracted when zpm deploy is run The help Section 13

18 14 Chapter 4. The zapp.yaml File

19 CHAPTER 5 Packaging This page is only interesting for project maintainers and packagers. This is not required for installing and using zpm. 1. Install Debian packaging dependencies: $ sudo apt-get install devscripts debhelper 2. Clone source from Git. Example: $ git clone 3. Amend the debian/changelog using dch. 4. Create a gzipped tarball of the zpm source (minus the debian/ dir): $ tar czf../zpm_0.1.orig.tar.gz * --exclude=debian Note that the.tar.gz file name will vary depending on the latest entry in the changelog. 5. Build a binary package: $ debuild or for a source package, $ debuild -S 15

20 16 Chapter 5. Packaging

21 CHAPTER 6 Glossary nexe A Native Client (NaCl) executable. This is a cross-compiled application that can be executed inside the sandbox created by ZeroVM. sandbox A secure execution environment created by ZeroVM. A nexe running inside a sandbox will not be able to access anything outside the sandbox. system image A system image is a pre-installed image that can be referenced in a job description. By referencing a system image, you avoid uploading a full tarball. 17

22 18 Chapter 6. Glossary

23 CHAPTER 7 API 7.1 ZPM Core Functions zpmlib.zpm.exec_table_header = [ Node, Status, Retcode, NodeT, SysT, UserT, DiskReads, DiskBytesR, Di Column labels for the execution summary table zpmlib.zpm.no_auth_msg = Auth version 1.0 requires ST_AUTH, ST_USER, and ST_KEY environment variables\nto be Message displayed if insufficient auth settings are specified, either on the command line or in environment variables. Shamelessly copied from python-swiftclient. zpmlib.zpm.system_map_zapp_path = boot/system.map path/filename of the system.map (job description) in every zapp class zpmlib.zpm.zerocloudconnection(authurl=none, user=none, key=none, retries=5, preauthurl=none, preauthtoken=none, snet=false, starting_backoff=1, max_backoff=64, tenant_name=none, os_options=none, auth_version= 1, cacert=none, insecure=false, ssl_compression=true, retry_on_ratelimit=false) An extension of the swiftclient.connection which has the capability of posting ZeroVM jobs to an instance of ZeroCloud (running on Swift). authenticate() Authenticate with the provided credentials and cache the storage URL and auth token as self.url and self.token, respectively. post_job(job, response_dict=none) Start a ZeroVM job, using a pre-uploaded zapp Parameters job (object) Job description. This will be encoded as JSON and sent to Zero- Cloud. zpmlib.zpm._add_file_to_tar(root, path, tar) Parameters root Root working directory. path File path. tar Open tarfile.tarfile object to add the files to. zpmlib.zpm._create_project_files(location, with_ui=false) Create a default zapp.yaml file in the specified directory location. If with_ui is True, add template UI files to location. 19

24 Raises a RuntimeError if any files would be overwritten in location. zpmlib.zpm._deploy_zapp(conn, target, zapp_path, auth_opts, force=false) Upload all of the necessary files for a zapp. Returns the name an uploaded index file, or the target if no index.html file was uploaded. Parameters force (bool) Force deployment, even if the target container is not empty. This means that files could be overwritten and could cause consistency problems with these objects in Swift. zpmlib.zpm._generate_job_desc(zapp) Generate the boot/system.map file contents from the zapp config file. Parameters zapp dict of the contents of a zapp.yaml file. Returns dict of the job description zpmlib.zpm._generate_uploads(conn, target, zapp_path, auth_opts) Generate sequence of (container-and-file-path, data, content-type) tuples. zpmlib.zpm._get_exec_table(resp) Build an execution summary table from a job execution response. Parameters resp (dict) Response dictionary from job execution. Must contain a headers key at least (and will typically contain status and reason as well). Returns Tuple of total execution time (str), prettytable.prettytable containing the summary of all node executions in the job. zpmlib.zpm._get_exec_table_data(headers) Extract a stats table from execution HTTP response headers. Stats include things like node name, execution time, number of reads/writes, bytes read/written, etc. Parameters headers (dict) dict of response headers from a job execution request. It must contain at least x-nexe-system, x-nexe-status, x-nexe-retcode, x-nexe-cdr-line. Returns Tuple of two items. The first is the total time for the executed job (as a str). The second is a table (2d list) of execution data extracted from X-Nexe-System and X-Nexe-Cdr-Line headers. Each row in the table consists of the following data: node name node time system time user time number of disk reads number of bytes read from disk number of disk writes number of bytes written to disk number of network reads number of bytes read from network number of network writes number of bytes written to network 20 Chapter 7. API

25 zpmlib.zpm._get_swift_zapp_url(swift_service_url, zapp_path) Parameters swift_service_url (str) The Swift service URL returned from a Keystone service catalog. Example: zapp_path (str) <container>/<zapp-file-name>. Example: test_container/myapp.zapp Here s a typical usage example, with typical input and output: >>> swift_service_url = ( AUTH_469a9cd20b5a4fc5be9438f66bb5ee04 ) >>> zapp_path = test_container/myapp.zapp >>> _get_swift_zapp_url(swift_service_url, zapp_path) swift://auth_469a9cd20b5a4fc5be9438f66bb5ee04/test_container/myapp.zapp zpmlib.zpm._guess_auth_version(args) Guess the auth version from first the command line args and/or envvars. Command line arguments override environment variables, so we check those first. Auth v1 arguments: --auth --user --key Auth v2 arguments: --os-auth-url --os-username --os-password --os-tenant-name If all of the v1 and v2 arguments are specified, default to 1.0 (this is how python-swiftclient behaves). If no auth version can be determined from the command line args, we check environment variables. Auth v1 vars: ST_AUTH ST_USER ST_KEY Auth v2 vars: OS_AUTH_URL OS_USERNAME OS_PASSWORD OS_TENANT_NAME The same rule above applies; if both sets of variables are specified, default to 1.0. If no auth version can be determined, return None. Parameters args argparse.namespace, representing the args specified on the command line ZPM Core Functions 21

26 Returns 1.0, 2.0, or None zpmlib.zpm._prepare_auth(version, args, conn) Parameters version (str) Auth version: 0.0, 1.0, or indicates no auth. args argparse.namespace instance, with attributes representing the various authentication parameters conn ZeroCloudConnection instance. zpmlib.zpm._prepare_job(tar, zapp, zapp_swift_url) Parameters Returns tar The application.zapp file, as a tarfile.tarfile object. zapp (dict) Parsed contents of the application zapp.yaml specification, as a dict. zapp_swift_url (str) Path of the.zapp in Swift, which looks like this: swift://auth_abcdef123/test_container/hello.zapp See _get_swift_zapp_url(). Extracted contents of the boot/system.map with the swift path to the.zapp added to the devices for each group. So if the job looks like this: [{ exec : { args : hello.py, path : file://python2.7:python }, devices : [{ name : python2.7 }, { name : stdout }], name : hello }] the output will look like something like this: [{ exec : {u args : hello.py, path : file://python2.7:python }, devices : [ { name : python2.7 }, { name : stdout }, { name : image, path : swift://auth_abcdef123/test_container/hello.zapp }, ], name : hello }] zpmlib.zpm.bundle_project(root) Bundle the project under root. zpmlib.zpm.create_project(location, with_ui=false) Create a ZeroVM application project by writing a default zapp.yaml in the specified directory location. Parameters location Directory location to place project files. with_ui Defaults to False. If True, add basic UI template files as well to location. Returns Full path to the created zapp.yaml file. zpmlib.zpm.execute(args) Execute a zapp remotely on a ZeroCloud deployment. 22 Chapter 7. API

27 Returns A dict with response data, including the keys status, reason, and headers. zpmlib.zpm.find_project_root() Starting from the cwd, search up the file system hierarchy until a zapp.yaml file is found. Once the file is found, return the directory containing it. If no file is found, raise a RuntimeError. zpmlib.zpm.render_zapp_yaml(name, template_name= zapp.yaml ) Load and render the zapp.yaml template ZPM Core Functions 23

28 24 Chapter 7. API

29 CHAPTER 8 Indices and tables genindex modindex search 25

30 26 Chapter 8. Indices and tables

31 Python Module Index z zpmlib.zpm, 19 27

32 28 Python Module Index

33 Index Symbols _add_file_to_tar() (in module zpmlib.zpm), 19 _create_project_files() (in module zpmlib.zpm), 19 _deploy_zapp() (in module zpmlib.zpm), 20 _generate_job_desc() (in module zpmlib.zpm), 20 _generate_uploads() (in module zpmlib.zpm), 20 _get_exec_table() (in module zpmlib.zpm), 20 _get_exec_table_data() (in module zpmlib.zpm), 20 _get_swift_zapp_url() (in module zpmlib.zpm), 20 _guess_auth_version() (in module zpmlib.zpm), 21 _prepare_auth() (in module zpmlib.zpm), 22 _prepare_job() (in module zpmlib.zpm), 22 A authenticate() (zpmlib.zpm.zerocloudconnection method), 19 B bundle_project() (in module zpmlib.zpm), 22 C create_project() (in module zpmlib.zpm), 22 E EXEC_TABLE_HEADER (in module zpmlib.zpm), 19 execute() (in module zpmlib.zpm), 22 F find_project_root() (in module zpmlib.zpm), 23 N nexe, 17 NO_AUTH_MSG (in module zpmlib.zpm), 19 P post_job() (zpmlib.zpm.zerocloudconnection method), 19 R render_zapp_yaml() (in module zpmlib.zpm), 23 S sandbox, 17 system image, 17 SYSTEM_MAP_ZAPP_PATH (in module zpmlib.zpm), 19 Z ZeroCloudConnection (class in zpmlib.zpm), 19 zpmlib.zpm (module), 19 29