OTC Tools Development and Release process Igor Stoppa & Eduard Bartosh & JF Ding V0.12 23 May 2013
Purpose of the document This document describes development and release workflow for the following projects: Tools releases: bmap-tools, depanneur, gbs, git-buildpackage and mic QA Tools: CATS, QA Reports. These are released independently. Services packages: Jenkins-jobs, Jenkins-plugins, Jenkins-scripts, and all the packages for BOSS. For the above mentioned projects it s strongly recommended to follow the workflow and rules described in this document. For other projects it s up to the project maintainers to decide to use the workflow and rules described in this document, use them partly or not to use at all. In the following of this document, all the references of Tools names are for the convenience of description, which can be replaced by other different component name, e.g. Services 2 INTEL CONFIDENTIAL
devel patch A patch B submit for review (Jenkins) pre-review build & test OBS build in sandbox Development & Release Process (Gerrit) human review & acceptance gerrit devel branch (Jenkins) upload (OBS) integration into <project>:devel build test OBS <project>:devel (Jenkins) publish Repos 1 2 3 4 5 rebase required Branching & version management tizen.org tzdev.org intranet OBS <project>:devel <project>:pre-release <project> 1) developers submit patches for review 2) preliminary automated verification: test packages are built in a sandbox and smoke tested 3) the submission is reviewed when accepted it requires rebase and resubmit for other changes queued for review 4) upload to OBS of the new version of the package(s) 5) the packages are published to the appropriate location 3 INTEL CONFIDENTIAL
Branching process 1. Developers submit changes to devel branch devel branch release-<rnumber> branch Master branch 1 2 3 4 6 5 7 8 2. When the devel branch meets the criteria for rolling out a release, it is branched into a release-<rnumber> branch 3. The release-<rnumber> undergoes a stabilization phase during which all changes submitted to release-<number> branch have to be submitted also to devel. 4. Release candidates should be tagged as v<rnumber>rc<num> 5. Eventually the branch is tagged as v<rnumber> and a release is made. 6. Master Branch is reset to the release tag and doesn t receive any direct commit. 7. Bugfix branch is made for current release patch tag & release 8. Master Branch is reset once more to the latest bugfix release. 4 INTEL CONFIDENTIAL
Versioning Recommended versioning scheme for releases is major.minor[.bugfix] Release candidates should add -rc<num> suffix to release version, for example: 1.2-rc1 or 1.2.3-rc1 In order to make package versions reflect versioning scheme and look similar for rpm and deb they should follow these recommendations: rpm.spec files: Release candidate: Version: 1.2, Release: 0.rc1.<CI_CNT>.<B_CNT> Final release: Version: 1.2, Release: 1.<CI_CNT>.<B_CNT> debian/changelog Release candidate: 1.2-0.rc1 Final release: 1.2-1 After creating release- branch package version in devel branch must be increased to ensure that version in <project>:devel repository is always higher than in <project>pre-release and <project> 5 INTEL CONFIDENTIAL
Advice of the Versioning for Services Packages Because we need not to release the services packages to end users currently, not like developer tools. And we need only to deploy them to the product infrastructure servers instead. So the versioning of these packages can be simpler: No rc versions No bugfixing versions Using <Major>.<Minor> versioning schema only, for every release, just increase the minor number. And for some important milestone release, increase the major number. 6 INTEL CONFIDENTIAL
Submitting changes Changes are submitted for review using git push <origin> <commit id>:refs/for/<target branch> Every submission triggers Jenkins job OTC-Tools-Tester-<project name>. This job does the following: Submits sources to OBS for the build to a project home:tester:tools-<project name>-<gerrit change number>.<gerrit patchset number>. OBS buld targets for the builds are taken from base project. For Tools releases it is Tools:Devel. Result packages uploaded to build repository at http://download.otctools.jf.intel.com/home%3a/tester%3a/tools-<package name>-<gerrit change number>.<gerrit patchset>/ Performs package installations in clean virtual environments for all target distributions. Optionally runs pylint and nosetests for python projects and produces violation, code coverage and test reports 7 INTEL CONFIDENTIAL
Accepting changes When change is accepted to the target branch(devel or release-<number>) in Gerrit OTC-Tools-Tester-<project name> is also triggered. For merged changes it does the same steps as on change submission, but upload sources to base OBS project. For devel branch it uploads sources to Tools:Devel For release-<number> branches it uploads sources to Tools:Pre-release This way system maintains consistency between code in devel and release branches and packages in correspondent OBS projects. It s recommended to configure Gerrit to allow only fast-forward merges. In this case changes have to be rebased on top of target branch before merging and pushed to Gerrit again. This will trigger OTC-Tools-Tester-<project name> job again and change will be tested again. It will reduce chances to get broken code to the target branches. 8 INTEL CONFIDENTIAL
Criteria for rejection of submissions the submission claims to fix a bug, but the bug is still reproducible with roughly the same frequency (if the frequency is significantly reduced we probably still want to accept it) the submission claims to implement a feature, but the feature verification fails so much that the feature is considered to be unusable (in practice the submission is not adding any value) the submission degrades the overall code quality (fails automated code analysis & verification / has problems with the implementation) the submission introduces major regressions (more on regression testing in separate foil) 9 INTEL CONFIDENTIAL
Regression testing Regression testing must be performed at least before releasing a tool. It should be done also periodically, during the development phase, preferably synchronised with integration of major changes. Specific test cases depend on the nature of each tool. Generic steps: take previous execution of the stable (production) release of the tool use the same data with the development release of the tool verify that the output is consistent and the only differences are due to bugfixes and planned features. 10 INTEL CONFIDENTIAL
Regression Testing in Staging Environment Because it s very critical for Tizen development process if we release and deploy the tools and services packages in product servers, we need to test them in staging environment before being deployed in product infrastructure servers. We d better host staging servers for all the three supporting domains: TZ, RSA and tizen.org, and testing the new packages in them accordingly. 11 INTEL CONFIDENTIAL
Smoke Testings for MIC Images For the regression testings for MIC, we need do simple verification for the outputs, which means the smoke testings for images. We need to confirm the output images can boot and work properly, with the same workable package repositories. We need to verify all the supported verticals: Mobile, PC, IVI, at least. 12 INTEL CONFIDENTIAL
Releasing Release stabilization is done in release-<rnumber> branches Jenkins job OTC-Tools-Tester-<project name> is taking care of synchronizing code in release-<rnumber> branches and packages in Tools:Pre-release Pre-release testers should install packages from http://download.otctools.jf.intel.com/home%3a/tester%3a/tools-<package name>-<gerrit change number>.<gerrit patchset>/ repos to verify changes under review and from http://download.otctools.jf.intel.com/tools%3a/prerelease/ to test accepted changes. When release is ready it s tagged with v<rnumber> tag and master branch is resetted to it. Packages from Tools:Pre-release are copied to Tools 13 INTEL CONFIDENTIAL
Bugfix releases (need to update it according current) Bugfix releases should be done in bugfix-<rnumber> branches. Release- <rnumber> branches can t be used for bugfix releases as they re mapped to Tools:Pre-release OBS project, which may contain packages scheduled for the next release. Jenkins job OTC-Tools-Tester-<project name> is taking care of synchronizing code in bugfix-<rnumber> branches and packages in Tools:Bugfix Bugfix testers install packages from http://download.otctools.jf.intel.com/home%3a/tester%3a/tools-<package name>- <gerrit change number>.<gerrit patchset>/ repos to verify changes under review and from http://download.otctools.jf.intel.com/tools%3a/bugfix/ to test accepted changes. When release is ready it s tagged with v<rnumber> tag and master branch is reset to it. Packages from Tools:Bugifx are copied to Tools 14 INTEL CONFIDENTIAL
Publishing releases and pre-releases Tools and Tools:Prerelease repositories are published to the http://download.tizen.org/tools/. Job is run by release managers manually. Job publishes repos to public location, extracts release notes from packages and updates.repo files with the proper urls. Similarly, Services and Services:prerelease repos will be published to http://download.tizen.org/tools/ci/. Package maintainers should put RELEASE_NOTES file into packages using %doc RELEASE_NOTES in spec file. Release notes will be extracted by the publishing job and uploaded to the public site as RELEASE_NOTES.<package name> Structure of the public download site contains archive of all releases, pre-release and link to the latest release. 15 INTEL CONFIDENTIAL
Development and release flow: Jenkins Git branch OBS download repo testing patch failed pre-review testing pass reject peer review accept Devel branch branch Tools:Devel download.otctools.jf.intel.com/tools:/devel/ Functional testing hot fix peer review accept release-* branch Tools:pre-release download.tizen.org/tools/pre-release/ Full testing merge Deploy request master branch Tools download.tizen.org/tools/archive/ download.tizen.org/tools/latest-release/ Sanity testing Release announce handled by Jenkins automatically 16 INTEL CONFIDENTIAL
Repository for Pre-announcement Deployment According our process, we need to deploy the new packages in all domains product servers. Need to use the latest published repo under archive for the deployment. After confirming everything is ok, and before the official announcement, need to update the repo URL to latest-release. 17 INTEL CONFIDENTIAL