lf-releng-global-jjb Release v dev3+b5f41d5

Transcription

1 lf-releng-global-jjb Release v dev3+b5f41d5 Jan 28, 2019

2

3 Contents 1 Release Notes 3 2 Guides Release Notes Install Configuration Best Practices Glossary Appendix Global JJB Templates C/C++ Jobs CI Jobs Global Macros Maven Jobs NodeJS Jobs OpenStack Heat Python Jobs ReadTheDocs Jobs Jenkins Views Indices and tables 75 i

4 ii

5 Linux Foundation Release Engineering Global Jenkins Job Builder (JJB) Documentation. Global-JJB is a library project containing reusable Jenkins Job Builder templates. Developed for LFCI to deploy management Jenkins jobs to an LF managed Jenkins instance, there are other jobs defined which may be helpful to projects that use the same build technology. The intention is to help projects save time from having to define their own job templates. Contents 1

6 2 Contents

7 CHAPTER 1 Release Notes Global JJB provides regular releases. The release notes for all releases are available in the relnotes directory in Git. 3

8 4 Chapter 1. Release Notes

9 CHAPTER 2 Guides 2.1 Release Notes v Bug Fixes Extend ${JOB_NAME} to include {java-version} parameter to support jobs to build with multiple versions of openjdk{8,11} v New Features Add a puppet-verify job to lf-ci-jobs. This job will perform Puppet linting on the specified repository. - project: name: lf-infra-puppet-mymodule project-name: lf-infra-puppet project: puppet/modules/mymodule jobs: - gerrit-puppet-verify Bug Fixes maven-fetch-metadata.sh was not respecting the -f (for file path) flag in MAVEN_PARAMS, causing lfmaven-merge jobs that utilize this flag to fail. It will now set a path based on this flag if it is present, or default to the current working directory. Check openjdk $VERSION before setting $JAVA_HOME. This enables jobs to pass openjdk10 or openjdk11 on CentOS 7 images to use the OpenJDK version installed in /opt. 5

10 2.1.3 v Bug Fixes Compress and upload all jjb-verify XML files to Nexus, to ease out the IOPs on cron jobs that manage the logs on Nexus and optimize job performace by ~8 mins. This is because the job generates around ~2.3K XML files (small files) which is uploaded to Nexus in every run of jjb-verify. Doing this is faster as compared to the Nexus Unpack plugin in the Nexus end unpacking the zip file we upload takes longer v Other Notes Update lftools version to v v New Features New lf-stack-create macro allows job-templates to setup a OpenStack Heat stack, useful for spinning up CSIT labs to run integration tests against. Use with the lf-stack-delete macro. Concurrency for the gerrit-jjb-verify job can now be configured by setting the build-concurrent parameter. New macro lf-maven-central is available to deploy artifacts to OSSRH staging for jobs that want to eventually deploy to Maven Central job-template: name: lf-maven-central-macro-test ##################### # Default variables # ##################### mvn-central: true ossrh-profile-id: '' ##################### # Job configuration # ##################### builders: - lf-maven-central: mvn-central: '{mvn-central}' ossrh-profile-id: '{ossrh-profile-id}' The GERRIT_REFSPEC build parameter can now be used to trigger a test build from the Jenkins Sandbox system against a work in progress packer image patch from a GitHub Pull Request. Upgrade Notes lf-stack-delete has been modified to be a companion macro to lf-stack-create in order to cleanup the stack at the end of a job run. It now includes a required parameter openstack-cloud to choose the clouds. 6 Chapter 2. Guides

11 yaml cloud configuration for the project. Existing users of this macro will need to update their job templates accordingly. Requires JJB for the jenkins-sandbox-cleanup job to not fail. Note: Despite the failure if JJB is not available the job will successfully delete all jobs and views, the primary purpose of this job. Bug Fixes RELENG-1450 All view disappears on Jenkins Sandbox after views are deleted. The All view is now recreated after delete-all is run v New Features Add the ability to configure the location of JJB s cache directory for CI jobs. New view-templates project-view, common-view, and csit-view are available for projects to manage Jenkins views through code. To use the project-view template in a project: - project: name: aaa-view views: - project-view project-name: aaa To use the common-view template in a project: - project: name: daily-builds views: - common-view view-name: Periodic view-regex: '.*-periodic-.*' To use the csit-view template in a project: - project: name: csit views: - csit-view view-name: CSIT view-regex: '.*csit.*' - project: name: csit-1node views: (continues on next page) 2.1. Release Notes 7

12 - csit-view (continued from previous page) view-name: CSIT-1node view-regex: '.*-csit-1node-.*' Add support to maven-stage jobs to publish to Maven Central via OSSRH. This is accomplished by adding these 2 new optional parameters to the job configuration. - gerrit-maven-stage: mvn-central: true ossrh-profile-id: 7edbe The openstack-cron job now has the ability to remove images older than a specified age (default: 30). The openstack-cron job now has the ability to remove orphaned servers. The openstack-cron job now has the ability to remove orphaned stacks. The openstack-cron job now has the ability to remove orphaned volumes. lf-infra-gerrit-scm and lf-infra-github-scm now require a submodule-timeout parameter to provide a timeout value (in minutes) for git fetch operations. All job-templates now provide an optional submodule-timeout parameter for git fetch operations, defaulting to 10 minutes. Upgrade Notes Some LF projects are already using a common-view template in their local ci-management repo. This common-view is called project-view in global-jjb so rename all instances of common-view to project-view when upgrading and remove the local common-view view-template definition from cimanagement. The openstack-cron job now requires a new parameter configured jenkins-urls in order to use the job. Any project using the lf-infra-gerrit-scm and lf-infra-github-scm macros in global-jjb should need to add a submodule-timeout value. It is recommended to default this value to 10 since it is the default used by the Jenkins Git Plugin. Bug Fixes The jenkins-init scripts dir is now updated to reflect changes recommended for the v release. We unfortunately missed this critical piece in the v release. Specify refspec to be blank for SCM config on github-maven-merge job. +refs/pull/*:refs/remotes/origin/pr/* causes there to be no merge job triggered. Setting the refspec to New detox version requires tox >= 3.5 and < 4. Instead of explicitly pulling tox, we are now implicitly pulling via detox. ( Other Notes lftools openstack module will now be installed as part of pre-build. The openstack-cron job now runs every hour instead of daily. This is because stack cleanup should happen more regularly. 8 Chapter 2. Guides

13 2.1.7 v New Features Add a new nexus-iq-namespace optional parameter to insert a namespace into Nexus IQ AppID. This is useful for shared Nexus IQ systems where projects might have concern about namespace collision. Note: We recommend when using the namespace to add a trailing - to the value. Eg. odl-, this is to make the namespace look nice for example odl-aaa is the result of namespace odl-, and project name aaa. Add lf-infra-publish-windows. A publisher for use at the end of Windows based job-templates. Bug Fixes Fix packer-verify job to correctly work with clouds.yaml config model implemented in global-jjb v v Bug Fixes jjb-cleanup.sh may be merged with other shell scripts that set -u which causes Jenkins to fail when activating virtualenvs v New Features Add support to the packer-build job to use clouds.yaml for openstack builder configuration rather than through the cloud-env file. This allows us to simplify the template configuration for openstack builders moving forward. New macro lf-sigul-sign-dir available to sign artifacts in a provided directory using Sigul. Usage: - lf-sigul-sign-dir: sign-dir: '$WORKSPACE/m2repo' This macro also requires a boolean variable to SIGN_ARTIFACTS to be set to true to activate the macro. We recommend the job-template that uses this macro to define it in the job parameters section. Example: - bool: name: SIGN_ARTIFACTS default: '{sign-artifacts}' description: Use Sigul to sign artifacts. Add Sigul signing support to the maven-staging job. sign-artifacts: true. Example: To activate Sigul signing make sure to set 2.1. Release Notes 9

14 - project: name: abc jobs: - gerrit-maven-stage sign-artifacts: true Add lf-stack-delete macro to delete an openstack heat stack at the end of the job. This macro requires a parameter defined in the job named STACK_NAME containing the name of the stack to delete. Add lf-infra-wrappers-windows to handle Windows specific wrapper configuration. Refactor lf-infra-wrappers to be for Linux systems and split out the non-linux specific components into a new lf-infra-wrappers-common. This change is seamless for current users of lf-infra-wrappers. Upgrade Notes Upgrade to global-jjb v before performing this upgrade. This ensures that jjb-verify job pulls in a regex fix that will allow it to verify the v upgrade. Global JJB now has non-jjb YAML configuration and requires action on the ci-management repo when upgrading to this version of Global JJB to prevent JJB from picking up these YAMLs as config. Follow the instructions below BEFORE upgrading globall-jjb: cd <git-root> git mv jjb/global-jjb global-jjb mkdir jjb/global-jjb ln -s../../global-jjb/shell jjb/global-jjb/shell ln -s../../global-jjb/jjb jjb/global-jjb/jjb git add jjb/global-jjb git commit -sm "Prepare repo for global-jjb v0.25.0" Minimum packer version is now required for the packer-build job. lf-infra-packer-build macro now requires 2 new variables to be passed. 1. openstack: Set to true if template is built using the openstack builder 2. openstack-cloud: The clouds.yaml cloud to use when running packer build Deprecation Notes lftools-install.sh is deprecated and will be removed in a future release. We recommend installing lftools via pip install user lftools to install instead of using this script. Bug Fixes Fix pip install pip setuptools which seems to fail against the Nexus 3 proxy. Run them as separate calls to make things happier. jjb-verify will now test on all changes in the jjb directory. The previous pattern was too specific and sometimes missed verifying patches that should be verified. Replace jjb-verify to test on all changes in the shell/* directory. 10 Chapter 2. Guides

15 Fix the lftools virtualenv workaround we had to put in place in the tox-verify job by using pip install --user for global tool installs. Fix jobs failing with UNSTABLE build due to install pip==18.0 missing. This change moves all the jobs to using lf-infra-pre-build to install lftools via user command. Use python -m pip to ensure that we are using the pip version that was installed rather than the OS wrapper version of pip. Fix package listing script in post-builder from causing UNSTABLE build due to difference in the two files being compared. Fix RTD job failing to find PBR install. Other Notes Update lftools to ~ Install global-jjb requires configuration in 2 places; Jenkins and the ci-management repository Jenkins configuration On the Jenkins side, we need to prep environment variables and plugins required by the jobs in global-jjb before we can start our first jobs. Install Jenkins plugins Install the following required Jenkins plugins and any optional ones as necessary by the project. Required Config File Provider Description Setter Environment Injector Plugin Git plugin Post Build Script SSH Agent Workspace Cleanup Required for Gerrit connected systems Gerrit Trigger Required for GitHub connected systems GitHub plugin GitHub Pull Request Builder Optional 2.2. Install 11

16 Mask Passwords MsgInject OpenStack Cloud Timestamper Environment Variables The Jenkins Configuration Merge job can manage environment variables job but we must first bootstrap them in Jenkins so that the job can run and take over. Required: JENKINS_HOSTNAME=jenkins092 NEXUS_URL= SILO=production SONAR_URL= Gerrit: GERRIT_URL= GitHub: GIT_URL= Note: Use GIT_CLONE_URL for GitHub projects as this will be different from the URL used in the properties configuration. Optional: LOGS_SERVER= Steps 1. Navigate to 2. Configure the environment variables as described above 3. Configure the same environment variables in the ci-management repo ci-management ci-management is a git repository containing JJB configuration files for Jenkins Jobs. Deploying Global JJB here as a submodule allows us easy management to install, upgrade, and rollback changes via git tags. Install Global JJB as follows: 1. Install Global JJB GLOBAL_JJB_VERSION=v0.1.0 git submodule add cd global-jjb (continues on next page) 12 Chapter 2. Guides

17 git checkout $GLOBAL_JJB_VERSION cd.. git add jjb/global-jjb (continued from previous page) # Setup symlinks mkdir jjb/global-jjb ln -s../../global-jjb/shell jjb/global-jjb/shell ln -s../../global-jjb/jjb jjb/global-jjb/jjb git add jjb/global-jjb git commit -sm "Install global-jjb $GLOBAL_JJB_VERSION" Note: We are purposely using github for production deploys of global-jjb so that uptime of LF Gerrit does not affect projects using global-jjb. In a test environment we can use global-jjb if desired. 2. Setup jjb/defaults.yaml Create and configure the following parameters in the jjb/defaults.yaml file as described in the defaults.yaml configuration docs <defaults-yaml>. Once configured commit the modifications: git add jjb/defaults.yaml git commit -sm "Setup defaults.yaml" 3. Push patches to Gerrit / GitHub using your favourite push method At this point global-jjb installation is complete in the ci-management repo and is ready for use Deploy ci-jobs The CI job group contains jobs that should deploy in all LF Jenkins infra. The minimal configuration to deploy the {project-name}-ci-jobs job group as defined in lf-ci-jobs.yaml is as follows: jjb/ci-management/ci-management.yaml: - project: name: ci-jobs jobs: - '{project-name}-ci-jobs' project: ci-management project-name: ci-management build-node: centos7-builder-2c-1g Required parameters: project The project repo as defined in source control. project-name A custom name to call the job in Jenkins. build-node The name of the builder to use when building (Jenkins label). Optional parameters: 2.2. Install 13

18 branch The git branch to build from. (default: master) jjb-version The version of JJB to install in the build minion. (default: <defined by the global-jjb project>) Deploy packer-jobs The packer job group contains jobs to build custom minion images. The minimal configuration needed to deploy the packer jobs is as follows which deploys the {project-name}-packer-jobs job group as defined in lf-ci-jobs.yaml. jjb/ci-management/packer.yaml: - project: name: packer-builder-jobs jobs: - '{project-name}-packer-jobs' project: ci-management project-name: ci-management branch: master build-node: centos7-builder-2c-1g platforms: - centos - ubuntu templates: builder - project: name: packer-docker-jobs jobs: - '{project-name}-packer-jobs' project: ci-management project-name: ci-management branch: master build-node: centos7-builder-2c-1g templates: docker platforms: - centos - ubuntu Required parameters: project The project repo as defined in source control. project-name A custom name to call the job in Jenkins. build-node The name of the builder to use when building (Jenkins label). platforms A list of supported platforms. templates A list of templates to build. We recommend setting one template per project section so that we can control which platforms to build for specific templates. Optional parameters: branch The git branch to build from. (default: master) 14 Chapter 2. Guides

19 packer-version The version of packer to install in the build minion, when packer is not available. (default: <defined by global-jjb>) 2.3 Configuration defaults.yaml This file lives in the ci-management repo typically under the path jjb/defaults.yaml. The purpose of this file is to store default variable values used by global-jjb templates. Required jenkins-ssh-credential The name of the Jenkins Credential to use for ssh connections. (ex: jenkins-ssh) lftools-version Version of lftools to install. Can be a specific version like or a PEP-440 definition For example <1.0.0 or >=1.0.0,< mvn-site-id Maven Server ID from settings.xml containing the credentials to push to a Maven site repository. mvn-staging-id Maven Server ID from settings.xml containing the credentials to push to a Maven staging repository. Gerrit required parameters: gerrit-server-name The name of the Gerrit Server as defined in Gerrit Trigger global configuration. (ex: Primary) GitHub required parameters: git-url Set this to the base URL of your GitHub repo. In general this should be If you are using GitHub Enterprise, or some other GitHub-style system, then it should be whatever your installation base URL is. This sets a job property that GitHub Pull Request Builder requires to work. Note that this is the web url to your project: (eg. \protect\t1\textdollarproject) git-clone-url This is the clone prefix used by GitHub jobs. Set this to either the same base url as git-url, or to git@github.com: including the trailing :. Determined by your clone method (https or git). github-org The name of the GitHub organization interpolated into the scm config. github_pr_org The name of the GitHub organization. All members of this organization will be able to trigger jobs. github_pr_whitelist List of GitHub members you wish to be able to trigger jobs. github_pr_admin_list List of GitHub members that will have admin privileges on the jobs. Example Gerrit Infra: - defaults: name: global # lf-infra defaults jenkins-ssh-credential: jenkins-ssh gerrit-server-name: OpenDaylight lftools-version: '<1.0.0' mvn-site-id: opendaylight-site mvn-staging-id: opendaylight-staging 2.3. Configuration 15

20 Example GitHub Infra: - defaults: name: global # lf-infra defaults jenkins-ssh-credential: jenkins-ssh github-org: lfit github_pr_whitelist: - jpwku - tykeal - zxiiro github_pr_admin_list: - tykeal lftools-version: '<1.0.0' mvn-site-id: opendaylight-site mvn-staging-id: opendaylight-staging Jenkins Files global-jjb makes use of the Jenkins Config File Provider plugin to provide some default configurations for certain tools. This section details the files to define in Jenkins Manage Files configuration. npmrc This file contains default npmrc configuration and lives in $HOME/.npmrc. Documentation for npmrc is available via the npm project. Required This file MUST exist. An empty file is acceptable if a proxy is not available for the project. type Custom file Create a Custom file with contents: registry = clouds-yaml Needed by openstack client and packer to fetch OpenStack credentials and configuration. This file is Open- Stack s clouds.yaml file. Optional Needed for jobs that use openstack client. packer if building against OpenStack infra. type Custom file Create a Custom file with contents: clouds: vex: auth: project_name: OS_PROJECT_NAME username: OS_USERNAME password: OS_PASSWORD auth_url: ' user_domain_name: Default (continues on next page) 16 Chapter 2. Guides

21 project_domain_name: Default region_name: ca-ymq-1 (continued from previous page) Warning: If using packer make sure that the clouds.yaml profile configuration is NOT configured. Using profile causes packer to look for another file called clouds-public.yaml for configuration. pipconf This file contains default configuration for the python-pip tool and lives in $HOME/.config/pip/pip.conf. Documentation for pip.conf is available via the pip project. Required This file MUST exist. An empty file is acceptable if a proxy is not available for the project. type Custom file Create a Custom file with contents: [global] timeout = 60 index-url = jjbini This file contains the Jenkins Job Builder configuration for CI Jobs. Required This file MUST exist. type Custom file Create a Custom file with contents: [job_builder] ignore_cache=true keep_descriptions=false include_path=.:scripts:~/git/ recursive=true [jenkins] user=jenkins-jobbuilder password= abcdef abcdef url= query_plugins_info=false [production] user=jenkins-jobbuilder password= abcdef abcdef url= query_plugins_info=false [sandbox] user=jenkins-jobbuilder password= abcdef abcdef url= query_plugins_info=false 2.3. Configuration 17

22 The last 2 sections are for the jenkins-cfg job use, they should match the silo names for the respective Jenkins systems, typically production and sandbox. jenkins-log-archives-settings See lf-infra-ship-logs for usage. If not archiving logs then keep this file with default settings, global-jjb needs the file to exist to function. Requires a credential named logs of type Username and Password created in the Jenkins Credentials system. 1. Add Server Credentials 2. Set ServerId to logs 3. Set Credentials to the logs user created in the Credentials System Required This file MUST exist if using log archiving. type Maven settings.xml <?xml version="1.0" encoding="utf-8"?> <settings xmlns=" xmlns:xsi=" xsi:schemalocation=" apache.org/xsd/settings xsd"> </settings> Note: This example is the default boilerplate generated by Jenkins with the comments stripped out. We can also use the default generated by Jenkins without modifying it. packer-cloud-env Cloud environment configuration variables for Packer jobs. whichever clouds packer jobs are using. Required This file MUST exist to use packer jobs. type Json file These can contain credentials and configuration for { } "cloud_auth_url": " "cloud_tenant": "TENANT_ID", "cloud_user": "CLOUD_USERNAME", "cloud_pass": "CLOUD_PASSWORD", "cloud_network": "CLOUD_NETWORK", "ssh_proxy_host": "" Jenkins CI Jobs jenkins-cfg-merge This job manages Jenkins Global configuration. Refer to the CI Documentation for job configuration details. 18 Chapter 2. Guides

23 2.3.4 Log Archiving The logs account requires a Maven Settings file created called jenkins-log-archives-settings with a server ID of logs containing the credentials for the logs user in Nexus. 2.4 Best Practices JJB YAML Layout Note: While some of this applies to the Global JJB project other recommendations are generally useful to projects that might be defining JJB templates. The Global JJB project is a useful example project to look at so we recommend referring to the Maven job definitions as an example as you read the documentation below: We recommend sectioning off the template into 3 general sections in order: 1. Job Groups (optional) 2. Common Functions 3. Job Template Definitions In section 1) not all configurations need this so is an optional section. Job groups are useful in cases where there are jobs that are generally useful together. For example the OpenDaylight uses a lot of Merge and Verify job combinations so every new project will want both job types defined in their project. In section 2) we want to define all common functions (anchors, aliases, macros) that are generally useful to all jobs in the file. This allows job template developers to look at the top of the file to see if there are useful functions already defined that they can reuse. In section 3) we can declare our job definitions. In the Global JJB project we create Gerrit and GitHub versions of the jobs so the format we use here might look strange at first but is well layed out for code reuse if we need to define 2 or more versions of the same job template for different systems. We will define this in more detail in the next section. Job Template Layout 1. Comment of Job Template Name 2. Macro containing build definition of the job a. Macro named after job b. Complete documentation of the job parameters c. Default parameters defined by the job d. Job configuration 3. job-template definition containing build triggers In section 1) we need to declare a in large comment text to identify the job section. In section 2) we declare the actual job definition. This is so that we have a single macro that we call in all the real job-template sections that is reusable and not duplicating any code. First we declare the macro as the job name. Then in 2.b) we provide the complete documentation of the job parameters this is so that we can link users of the job to this file and they can understand fully what options they can configure for this particular job. Then we define defaults for any parameters that are optional. The last section we define the job configuration which completes the macro Best Practices 19

24 In section 3) we declare the actual job-template. Because of all the preparations above job template definitions should be small and simple. It needs to define the scm and job triggers. The Global JJB project needs to support both Gerrit and GitHub versions of the same job so the job definitions there have 2 templates for each job defined Passing parameters to shell scripts There are 2 ways to pass parameters into scripts: 1. JJB variables in the format {var} 2. Environment variables in the format ${VAR} We recommend avoiding using method 1 (Pass JJB variables) into shell scripts and instead always use method 2 (Environment variables). This makes troubleshooting JJB errors easier and does not require escaping curly braces. This method requires 3 steps: 1. Declare a parameter section or inject the variable as properties-content. 2. Invoke the shell script with include-raw-escape instead of include-raw. 3. Use the shell variable in shell script. The benefit of this method is that parameters will always be at the top of the job page and when clicking the Build with Parameters button in Jenkins we can see the parameters before running the job. We can review the parameters retroactively by visiting the job parameters page job/lastsuccessfulbuild/parameters/. Injecting variables as properties-content makes the variable local to the specific macro, while declaring it as parameter makes the variable global. Note: When a macro which invokes a shell script has no JJB parameters defined!include-raw-escape will insert extra curly braces, in such cases its recommended to use!include-raw Shell scripts When developing shell scripts for JJB we recommend to create shell scripts as a separate file instead of inlining in YAML. This way we can ensure that the ShellCheck linter can catch potential issues with the scripts. When writing the script itself, we recommend to redeclare all expected inputs at the top of the file using lowercase variable names before setting set -u after the inputs section. This ensures that all variables the script expects are at the top of the file which is useful for others to review and debug the script at a later stage. The set -u configuration before the start of the script code ensures that we catch any of these undeclared variables at the top of the file. Example: #!/bin/bash # Inputs tox_dir="${tox_dir:-$workspace}" tox_envs="${tox_envs:-}" # Script start set -eux -o pipefail #... script code goes here 20 Chapter 2. Guides

25 2.4.4 Usage of config-file-provider When using the config-file-provider plugin in Jenkins to provide a config file. We recommend using a macro so that we can configure the builder to remove the config file as a last step. This ensures that credentials do not exist on the system for longer than it needs to. ship-logs example: - builder: name: lf-ship-logs builders: - config-file-provider: files: - file-id: jenkins-log-archives-settings variable: SETTINGS_FILE - shell:!include-raw: -../shell/logs-get-credentials.sh - shell:!include-raw: -../shell/logs-deploy.sh - shell:!include-raw: -../shell/logs-clear-credentials.sh - description-setter: regexp: '^Build logs:.*' In this example the script logs-deploy requires a config file to authenticate with Nexus to push logs up. We declare a macro here so that we can ensure that we remove credentials from the system after the scripts complete running via the logs-clear-credentials.sh script. This script contains 3 basic steps: 1. Provide credentials via config-file-provider 2. Run logs-deploy.sh 3. Remove credentials provided by config-file-provider Preserving Objects in Variable References JJB has an option to preserve a data structure object when you want to pass it to a template. org/infra/jenkins-job-builder/definition.html#variable-references One thing that is not explicitly covered is the format of the variable name that you pass the object to. When you use the {obj:key} notation to preserve the original data structure object, it will not work if the variable name has a dash - in it. The standard that we follow, and recommend, is to use an underscore _ instead of a dash. Example:.. literalinclude:: _static/github-pr-trigger.example In the above example note the use of underscores in github_pr_whitelist, github_pr_admin_list, and github_included_regions Using single quotes around variables Its recommended to use single quotes around JJB variables {variable}-field during variable substitution or when using a variable in a string field, in other cases its recommended to drop the single quotes. Example: 2.4. Best Practices 21

26 - builder: name: lf-user-logs builders: - inject: properties-content: 'HOME={user-home}' - build-file: settings: '{settings-file}' file-version: '{file-version}' Variable expansion and Defaults JJB has a concept called Defaults which is what JJB will replace a variable with if unset. Variables can configure dynamic content in job-template sections and allow certain options in these sections to be configurable. The section that expands Defaults is Job Templates no other sections will expand a default. This documentation will explain how variables and defaults expansion works and which take precedence in JJB s variable expansion logic for the following configuration sections. macro job-template project default Macro sections Macro sections can contain variables but do NOT support default values getting filled in both at the macro definition level and at the defaults configuration level. Macros and Job Templates can use Macros but any variables defined in a Macro needs to pass a value or a new variable redefined in the Job Template if you want to pass on the configuration. So for example if you have a macro that has a {msg} variable: Example: - builder: name: echo-msg builders: - shell: "echo {msg}" Any downstream job-templates or macros that use this macro MUST pass in a msg: Hello definition or redefine the msg variable msg: {msg}. Job Template sections Job Template sections can use defaults in two ways. 1. Configure the message: - job-template: name: echo-hello-world builders: - echo-msg: msg: 'Hello World' 22 Chapter 2. Guides

27 2. Re-define {msg} variable - job-template: name: echo-message builders: - echo-msg: msg: '{message}' In option 2, we redefine the variable msg as {message} which a user of the job-template can now pass into the job their own custom message which is different than option 1, where we set a static message to pass in. We purposely redefined the {msg} to {message} here to show that you do not need to redefine it with the same name but we could have used the same name {msg} in the template too if we wanted to keep it the same. Job Templates can also default a default variable for the variables it defines. Example: - job-template: name: echo-message message: 'Hello World' builders: - echo-msg: msg: '{message}' This creates a job template variable called {message} which will default to Hello World if the user of the template does not explicitly pass in a message. We should be aware of 2 Defaults concepts: 1. Default as defined in the job-template 2. Default as defined in a defaults configuration (typically defaults.yaml) In this case there is a default {message} set in the job-template. JJB will use this default if the user (project section) does not declare a {message}. If we do not declare a default in the job-template then JJB will fallback to checking the defaults configuration. This means that the precedence of defaults is as follows: 1. User-provided 2. Job Template 3. Defaults.yaml Project sections Project sections define real jobs and pass in variables as necessary. Projects sections do NOT expand defaults.yaml. So you cannot configure a setting with {var} in here and expect defaults.yaml to fill it in for you. Define required configuration here. Example: - project name: foo jobs: - 'echo-message' message: 'I am foo' 2.4. Best Practices 23

28 Defaults sections Defaults sections are the absolute last thing JJB checks if a variable is not configured in a job-template and user did not pass in the variable. JJB will fill in whatever is in the defaults configuration. Variable expansion order of precedence seems to be: 1. job-group section definition 2. project section definition 3. job-template variable definition 4. defaults.yaml variable definition Note: Defaults set variables in job-templates and are NOT used in Macros. global-jjb should not provide job-group definitions and leave it up to users of global-jjb to create their own as a jobgroup as a variable defined in a job group the highest precedence. Global JJB should strive to be purely a job-template and macro library for downstream consumers. Final thoughts For any Basic Job Configuration for example concurrent, jdk, node etc... we cannot set defaults with the same name as JJB will not expand them. To use node we need to give the variable for that setting a different name such as build-node instead, if we want JJB to perform expansion for those settings. This issue affects top level job configuration, it does not appear to affect items below the top level such as calling a builder, wrapper or parameter. 2.5 Glossary ciman Short for ci-management. ci-management Refers to the SCM repository containing the JJB configuration files. In most LF Projects this is the repository named ci-management, but is releng/builder in the OpenDaylight project and releng in the OPNFV project. JJB Short for Jenkins Job Builder (JJB) a tool used to convert YAML definitions into XML as a way to define Jenkins job configuration. 2.6 Appendix ShellCheck When using ShellCheck to lint global-jjb or any projects that include global-jjb as part of their project (common with ci-management repos) then we require version 0.4.x of ShellCheck installed on the build vms. This version introduces annotations used by shell scripts in this repo. 24 Chapter 2. Guides

29 CHAPTER 3 Global JJB Templates Job template code is in the jjb/ directory but documentation is in the docs/jjb/ directory of this project. 3.1 C/C++ Jobs Job Templates CMake Stage Stage job which runs cmake && make && make install and then packages the project into a tar.xz tarball to produce a release candidate. Template Names {project-name}-cmake-stage-{stream} gerrit-cmake-stage github-cmake-stage Comment Trigger stage-release Required parameters build-node The node to run build on. jenkins-ssh-credential Credential to use for SSH. (Configure in defaults.yaml) nexus-group-id The Maven style Group ID for the namespace of the project in Nexus. staging-profile-id The unique Nexus Staging Profile ID for the project. Contact your infra admin if you do not know it. Optional parameters branch Git branch to fetch for the build. (default: master) build-days-to-keep Days to keep build logs in Jenkins. (default: 7) 25

30 build-dir Directory to build the project in. (default: $WORKSPACE/target) build-timeout Timeout in minutes before aborting build. (default: 60) cmake-opts Parameters to pass to cmake. (default: ) git-url URL clone project from. (default: $GIT_URL/$PROJECT) install-prefix CMAKE_INSTALL_PREFIX to use for install. (default: $BUILD_DIR/output) install-prefix: #!/bin/bash echo "Hello World." Listing 1: Example make-opts Parameters to pass to make. (default: ) pre-build Shell script to run before performing build. Useful for setting up dependencies. (default: ) stream Keyword that to represent a release code-name. Often the same as the branch. (default: master) submodule-recursive Whether to checkout submodules recursively. (default: true) submodule-timeout Timeout (in minutes) for checkout operation. (default: 10) version (default: ) Project version to stage release as. There are 2 methods for using this value: 1. Defined explicitly here. 2. Leave this value blank and set /tmp/artifact_version Use method 2 in conjunction with pre-build configuration to generate the artifact_version automatically from files in the project s repository. For example with pre-build script: #!/bin/bash MAJOR_VERSION="$(grep 'set(ocio_version_major' CMakeLists.txt awk '{{print $NF}}' awk -F')' '{{print $1}}')" MINOR_VERSION="$(grep 'set(ocio_version_minor' CMakeLists.txt awk '{{print $NF}}' awk -F')' '{{print $1}}')" PATCH_VERSION="$(grep 'set(ocio_version_patch' CMakeLists.txt awk '{{print $NF}}' awk -F')' '{{print $1}}')" echo "${{MAJOR_VERSION}}.${{MINOR_VERSION}}.${{PATCH_VERSION}} " > /tmp/artifact_version CMake Verify Verify job which runs cmake && make && make install to test a project build.. Template Names {project-name}-cmake-verify-{stream} gerrit-cmake-verify github-cmake-verify Comment Trigger recheck reverify 26 Chapter 3. Global JJB Templates

31 Required parameters build-node The node to run build on. jenkins-ssh-credential Credential to use for SSH. (Configure in defaults.yaml) Optional parameters branch Git branch to fetch for the build. (default: master) build-days-to-keep Days to keep build logs in Jenkins. (default: 7) build-dir Directory to build the project in. (default: $WORKSPACE/target) build-timeout Timeout in minutes before aborting build. (default: 60) cmake-opts Parameters to pass to cmake. (default: ) git-url URL clone project from. (default: $GIT_URL/$PROJECT) install-prefix CMAKE_INSTALL_PREFIX to use for install. (default: $BUILD_DIR/output) install-prefix: #!/bin/bash echo "Hello World." Listing 2: Example make-opts Parameters to pass to make. (default: ) pre-build Shell script to run before performing build. Useful for setting up dependencies. (default: ) stream Keyword that to represent a release code-name. Often the same as the branch. (default: master) submodule-recursive Whether to checkout submodules recursively. (default: true) submodule-timeout Timeout (in minutes) for checkout operation. (default: 10) gerrit_verify_triggers Override Gerrit Triggers. gerrit_trigger_file_paths Override file paths which to filter which file modifications will trigger a build. 3.2 CI Jobs Job Groups Job groups are a great tool to configure categories of jobs together at the same time. Below the example are some starting point job-groups but we recommend creating your own to ensure that the jobs configured reflect the project s needs. An example project: - job-group: name: odl-maven-jobs jobs: - gerrit-maven-clm (continues on next page) 3.2. CI Jobs 27

32 - gerrit-maven-merge - gerrit-maven-release - gerrit-maven-verify - gerrit-maven-verify-dependencies: build-timeout: 180 (continued from previous page) mvn-version: mvn35 - project: name: aaa jobs: - odl-maven-jobs In this example we are using the job-group to assign a list of common jobs to the aaa project. The job-group also hardcodes mvn-version to mvn35 and build-timeout to 180 for all projects using this job-group. A benefit of this method is for example disabling entire category of jobs by modifying the job-group, insert disable-job: true parameter against the jobs to disable. Below is a list of CI job groups: job-group: name: '{project-name}-ci-jobs' jobs: - gerrit-jenkins-cfg-merge - gerrit-jenkins-sandbox-cleanup - gerrit-jjb-deploy-job - gerrit-jjb-merge - gerrit-jjb-verify - job-group: name: '{project-name}-github-ci-jobs' jobs: - github-jenkins-cfg-merge - github-jenkins-sandbox-cleanup - github-jjb-deploy-job - github-jjb-merge - github-jjb-verify - job-group: name: '{project-name}-info-yaml-jobs' jobs: - gerrit-info-yaml-verify - job-group: name: '{project-name}-github-info-yaml-jobs' jobs: - github-info-yaml-verify - job-group: name: '{project-name}-packer-jobs' (continues on next page) 28 Chapter 3. Global JJB Templates

33 jobs: - gerrit-packer-merge - gerrit-packer-verify (continued from previous page) - job-group: name: '{project-name}-github-packer-jobs' jobs: - github-packer-merge - github-packer-verify Macros lf-infra-jjb-parameters Required Parameters jjb-cache Location of Jenkins Job Builder (JJB) cache used for jjb jobs. jjb-version Version of Jenkins Job Builder (JJB) to install and use in the jjb jobs. lf-jenkins-cfg-clouds Deploys Jenkins Cloud configuration read from the jenkins-clouds directory in ci-management repositories. Note: Requires the jjbini file in Jenkins CFP to contain JJB 2.0 style config definitions for production and sandbox systems. Required Parameters jenkins-silos Space-separated list of Jenkins silos to update configuration for as defined in ~/.config/jenkins_jobs/jenkins_jobs.ini lf-jenkins-cfg-global-vars Manages the Global Jenkins variables. This macro will clear all exist macros in Jenkins and replaces them with the ones defined by the ci-management/jenkins-config/global-vars-silo.sh script. Note: Requires the jjbini file in Jenkins CFP to contain JJB 2.0 style config definitions for production and sandbox systems. Required parameters jenkins-silos Space-separated list of Jenkins silos to update configuration for as defined in ~/.config/jenkins_jobs/jenkins_jobs.ini lf-infra-jjbini Provides jenkins_jobs.ini configuration for Jenkins CI Jobs 29

34 lf-infra-jjbini-sandbox Provides jenkins_jobs.ini configuration for Jenkins sandbox. Todo: This needs to be consolidated into lf-infra-jjbini when JJB 2.0 is available lf-packer-common Common packer configuration. lf-packer-file-paths Gerrit file-paths for packer jobs. lf-packer-parameters Parameters useful for packer related tasks. Parameters packer-version Version of packer to install / use. (shell: PACKER_VERSION) lf-packer-verify-file-paths Gerrit file-paths for packer verify jobs. lf-puppet-parameters Parameters useful for Puppet related tasks. Parameters puppet-lint-version Version of puppet-lint to install / use. (shell: PUP- PET_LINT_VERSION) Job Templates Gerrit Branch Lock Job submits a patch to lock or unlock a project s branch. Template Names {project-name}-gerrit-branch-lock-{stream} gerrit-branch-lock Comment Trigger lock branch unlock branch 30 Chapter 3. Global JJB Templates

35 Required parameters build-node The node to run build on. jenkins-ssh-credential Credential to use for SSH. (Generally should be configured in defaults.yaml) Optional parameters branch Git branch to build against. (default: master) git-url URL to clone project from. (default: $GIT_URL/$GERRIT_PROJECT) stream Keyword that can be used to represent a release code-name. Often the same as the branch. (default: master) submodule-timeout Timeout (in minutes) for checkout operation. (default: 10) gerrit_merge_triggers Override Gerrit Triggers. Jenkins Configuration Merge Jenkins job to manage Global Jenkins configuration. Note: Requires the jjbini file in Jenkins CFP to contain JJB 2.0 style config definitions for production and sandbox systems. Template names {project-name}-jenkins-cfg-merge gerrit-jenkins-cfg-merge github-jenkins-cfg-merge Optional parameters branch Git branch to build against. (default: master) cron How often to run the job on a cron schedule. git-url URL to clone project from. (default: $GIT_URL/$GERRIT_PROJECT) jenkins-silos Space separated list of Jenkins silos to update configuration for as defined in ~/.config/jenkins_jobs/jenkins_jobs.ini (default: production sandbox) Typically this template is automatically pulled in by the {project-name}-ci-jobs job-group and does not need to be explicitly called if the job group is being used. Minimal Example: project: name: jenkins-cfg-merge-minimal-test jobs: - 'gerrit-jenkins-cfg-merge' project-name: ci-management Full Example: 3.2. CI Jobs 31

36 --- - project: name: jenkins-cfg-merge-full-test jobs: - 'gerrit-jenkins-cfg-merge' project-name: builder jenkins-silos: releng sandbox Global Environment Variables Global Environment Variables are managed via the jenkins-config/global-vars-silo.sh file in cimanagement. Replace SILO with the name of the Jenkins silo the variable configuration is for. The format for this file is KEY=value for example: GERRIT_URL= GIT_BASE=git://devvexx.opendaylight.org/mirror/$PROJECT GIT_URL=git://devvexx.opendaylight.org/mirror JENKINS_HOSTNAME=vex-yul-odl-jenkins-2 LOGS_SERVER= NEXUS_URL= ODLNEXUSPROXY= SILO=sandbox SONAR_URL= Cloud Configuration This configuration requires the OpenStack Cloud plugin in Jenkins and is currently the only cloud plugin supported. OpenStack Cloud plugin version supported: Cloud configuration are managed via a directory structure in ci-management as follows: jenkins-config/clouds/openstack/ jenkins-config/clouds/openstack/cattle/cloud.cfg jenkins-config/clouds/openstack/cattle/centos7-builder-2c-2g.cfg jenkins-config/clouds/openstack/cattle/centos7-builder-4c-4g.cfg jenkins-config/clouds/openstack/cattle/centos7-docker-4c-4g.cfg The directory name inside of the openstack directory is used as the name of the cloud configuration. In this case cattle is being used as the cloud name. The cloud.cfg file is a special file used to configure the main cloud configuration in the format KEY=value. Cloud Parameters CLOUD_URL API endpoint URL for Keystone. (default: ) CLOUD_IGNORE_SSL Ignore unverified SSL certificates. (default: false) CLOUD_ZONE OpenStack region to use. (default: ) 32 Chapter 3. Global JJB Templates

37 CLOUD_CREDENTIAL_ID Credential to use for authentication to OpenStack. (default: os-cloud ) INSTANCE_CAP Total number of instances the cloud will allow spin up. (default: null) SANDBOX_CAP Total number of instances the cloud will allow to spin up. This applies to sandbox systems and overrides the INSTANCE_CAP setting. (default: null) Template Parameters Note: In the case of template definitions of a parameter below is not passed the one defined in default clouds will be inherited. IMAGE_NAME The image name to use for this template. (required) HARDWARE_ID OpenStack flavor to use. (required) LABELS Labels to assign to the vm. (default: FILE_NAME) NETWORK_ID OpenStack network to use. (default: ) USER_DATA_ID User Data to pass into the instance. (default: jenkins-init-script) INSTANCE_CAP Total number of instances of this type that can be launched at one time. When defined in clouds.cfg it defines the total for the entire cloud. (default: null) SANDBOX_CAP Total number of instances of this type that can be launched at one time. When defined in clouds.cfg it defines the total for the entire cloud. This applies to sandbox systems and overrides the INSTANCE_CAP setting. (default: null) FLOATING_IP_POOL Floating ip pool to use. (default: ) SECURITY_GROUPS Security group to use. (default: default ) AVAILABILITY_ZONE OpenStack availability zone to use. (default: ) START_TIMEOUT Number of milliseconds to wait for the agent to be provisioned and connected. (default: ) KEY_PAIR_NAME SSH Public Key Pair to use for authentication. (default: jenkins) NUM_EXECUTORS Number of executors to enable for the instance. (default: 1) JVM_OPTIONS JVM Options to pass to Java. (default: ) FS_ROOT File system root for the workspace. (default: /w ) RETENTION_TIME Number of minutes to wait for an idle slave to be used again before it s removed. If set to -1, the slave will be kept forever. (default: 0) CONNECTION_TYPE The connection type for Jenkins to connect to the build minion. Valid options: JNLP, SSH. (default: SSH ) For a live example see the OpenDaylight project jenkins-config directory. releng-builder/tree/master/jenkins-config Troubleshooting Cloud Configuration The directory groovy-inserts contains the groovy script output that is used to push to Jenkins. In the event of a job failure this file can be inspected CI Jobs 33