conda-forge Documentation

Transcription

1 conda-forge Documentation Release 0.0.1a1 conda-forge Oct 16, 2018

2

3 Contents 1 News and announcements Pinned Announcements Announcements Guidelines Transferring to conda-forge Intended Usage Renaming Packages Fixing Broken Packages Creating Conda Recipes Getting Started Step-by-step Instructions What happens after the PR to staged-recipes is merged Avoid Dependencies Outside of Conda-Forge Optional: bld.bat and/or build.sh Maintainer Role Other Recipes in staged-recipes Activate scripts Writing the meta.yaml Build from Tarballs, Not Repos Packaging the License Manually Populating the hash Field Excluding a Platform Pinning packages Using conda-build s run_exports feature Build matrices Compilers Building Against NumPy Building noarch packages Build Number Use pip Downloading extra sources and data files BLAS & Numpy & Friends 21 i

4 6 Building on Windows Simple CMake-Based bld.bat Building for different VC versions Testing in Recipes Simple Existence Tests Testing Python Packages Docker Tests Should a Recipe Run All of a Package s Tests? CI Services Travis CI (OS X) CircleCI (Linux, OSX) conda-smithy rerender How to re-render? When should we re-render? conda-forge gotchas Using multiple channels Using a fork vs a branch when updating a recipe Automatic services 35 please rerender please add noarch: python please update for conda-build please lint please update circle please update team Joining the team! staged-recipes Responsibilities TODO Governance Code of Conduct Teams & Roles Sub-Teams Voting Current Members of Core Document History Conda-Forge Subteams Bot Sub-Team Indices and tables 45 ii

5 Contents: Contents 1

6 2 Contents

7 CHAPTER 1 News and announcements 1.1 Pinned Announcements Deprecation notice for Python 3.5 As we start building out more of the python 3.7 stack, we will no longer be building python 3.5 packages. No new python 3.5 packages will be built after The compiler migration is in full swing. The bot will be making the rounds and modernizing more than 4000 packages. This is going to take a few months to get done so bear with us. 1.2 Announcements The rebuild is moving along nicely with almost a third of packages completed. Recently completed are numpy and openblas which should open up much of the python numeric stack. We re only about 5 feedstocks away from opening up all of R as well A minimal python 3.7 build is now available across all platforms and both compilers! Conda forge now has a magical status bar for tracking the progress of migrations. You can find this at conda-forge.org/status. 3

8 4 Chapter 1. News and announcements

9 CHAPTER 2 Guidelines 2.1 Transferring to conda-forge This document intends to layout some guidelines on the transfer of conda-recipes and anaconda-recipes to conda-forge. These aren t hard and fast rules and are open resonable interpretation and reviewer judegement. It is the aspiration that almost all recipes from those repos shall be proposed for addition here, thought it may be decided that a few don t actually belong or should not be supported anymore. When adding a package from either location, inspect the commit history to see who has made changes to the recipe in the past. Anyone who has touched the recipe should be pinged about it getting added here. They should also be consulted with regards to whether the recipe needs any tweaks before being added. Also, they should be asked if they would like to be added as maintainers. Only if they give express permission to be adding as a maintainer should they be added to the maintainer list. In all cases when porting a recipe, you should add yourself as a maintainer. Some contributors to these repos were quite prolific, but may not be as actively engaged. If they specify they only want to be contacted for certain recipes or none at all, please respect their wishes also add make note of what recipes (if any) they would like to be notified for. If they are no longer interested in any conda recipes, make that note here as well. Before contacting anyone please consult the list in this issue to see if that contributor has restrictions. When porting make sure the recipe follows the linting specifications. The section order should go package, source, build, requirements, test, about, extra/recipe-maintainers. It is recommended to add a build section with the number set to 0 explicitly even if the rest is unneeded. If there is no build for Windows, make sure to add skip: True # [win] to the build section. The about section must have the home URL (verify the URL is still correct), license (verify the correct license is present), and a one sentence (or few word) summary. When specifying the version it is strongly recommend that jinja templating be used to set the version at the top (e.g. {% set version = "0.10.1" %}) and then replace all uses of the version with {{ version }}. Preference should be given to compressed source balls as opposed to version control checkouts. Make sure all links to compressed source balls allow for easy changing of the version (using latest is not acceptable). Also, a checksum should be included with all compressed source balls to allow for verification of downloads. It is required to add tests with all packages. These can included but are not limited to checking libraries are installed, python imports, simple code snippet to compile or run a basic test, command line usage (checking help or version). It is suggested that compiled code run all tests (e.g. make check) to ensure it was built properly. This normally should happen in the build. 5

10 It is possible, though not recommended, to include multiple recipes into a single pull request on staged-recipes. condabuild-all is used to determine the build order and the necessary build matrix (e.g. which python versions to build against). From a practical perspective, there are limitations on the continuous integration resources and also on what reviewers are able/willing to review in a single pull request. A large pull request with many recipes makes it more difficult to review. If the recipes make it through these two constraints and is merged, race conditions amongst the different feedstocks may require work by you and/or core maintainers to restart them in such a way to build everything in a suitable order. None of this is to say that one can t add multiple recipes in a single pull request. One certainly can do this and it can work, but the recommendation is to open a PR with one recipe first, and to to ask for agreement about adding one or two additional recipes. 2.2 Intended Usage Maintainers time and CI resources are what enable conda-forge. They are as scarce as they are valuable. conda-forge has enough capacity to support releasing packages, but not developing them. Publishing a package to conda-forge signals it is suitable for users not involved with development. However, publishing does not always happen error-free. Multiple commits are acceptable when debugging issues with the release process itself. Fortunately, there are options for optimizing the development of a package. conda-smithy is a tool used by conda-forge itself to manage feedstocks. conda-smithy can be used to create an internal development feedstock that is seperate from conda-forge. ci-helpers is a set of scripts that drive various CI services using environment variables. 2.3 Renaming Packages Sometimes packages are misnamed. To correct the name of the package, please submit a PR into staged-recipes with the correct name. During the review process please make certain to note that the package is a rename and contact a member of conda-forge/core to remove the old feedstock (and potentially package if needed). 2.4 Fixing Broken Packages Sometimes you need to remove a package from the conda-forge channel on Anaconda.org. The reasons for this are many, but the ones that immediately come to mind are: Incorrect pinnings. This can cause the conda solver to preferentially pick up the old version of the package even though you ve bumped the build number. Packages renames We prefer to not remove packages for the following reasons: 1. Unaffected users are unable to get the broken package. 2. Unable to reverse (what if we were incorrect to remove it). 3. Loss of reproducibility (unable to create an old environment). 4. Not as community friendly (leaves no opportunity to review decision). 5. Blocks anyone from inspecting the broken packages. 6 Chapter 2. Guidelines

11 Though, we can easily address all of these by using a label for broken packages. If one simply drops the main label and replaces it with broken. This still makes it unavailable to install by default. However, the package remains available so as to avoid the problems listed above. To get your package relabeled as broken instead of main, please post on the issue tracker for your feedstock and with this request: Please move my package from ``main`` to ``broken`` on the conda-forge channel on anaconda.org because: <insert your reason> And also post to the conda-forge gitter channel! Also please be patient with us if we do not repond immediately :) 2.4. Fixing Broken Packages 7

12 8 Chapter 2. Guidelines

13 CHAPTER 3 Creating Conda Recipes This document presents conda-forge information about creating recipes overall. Details of modifying files inside of a recipe are present on other pages in this documentation. 3.1 Getting Started There are multiple ways to get started: 1. Look at the example recipe in the staged-recipes repository and modify it as necessary. 2. If it is an R package from CRAN, please instead start by using the conda-forge helper script for R recipes. Then if necessary you can make manual edits to the recipe. 3. If it is a python package you can generate a skeleton as a starting point with conda skeleton pypi your_package_name. You do not have to use the skeleton, and the recipes produced by skeleton will need to be edited. In particular, you ll at least need to change the build line to use pip, add yourself as a maintainer, and specify a license_file. Your final recipe should have no comments (unless they re actually relevant to the recipe, and not generic instruction comments), and follow the order in the example. If there are details you are not sure about please open a pull request. The conda-forge team will be happy to answer your questions. In case you are building your first recipe using conda-forge, a step-by-step instruction and checklist that might help you with a successful build is provided in the following. 3.2 Step-by-step Instructions 0. Ensure your source code can be downloaded as a single file. Source code should be downloadable as an archive (.tar.gz,.zip,.tar.bz2,.tar.xz) or tagged on GitHub, to ensure that it can be verified. (For further detail, see Build from Tarballs, Not Repos). 9

14 1. Fork the example recipes repository. 2. Create a new branch from the staged-recipes master branch. 3. Within your forked copy, generate a new folder in the recipes subdirectory and copy the meta.yml file from the example directory. Please leave the example directory unchanged! 4. Edit the copied recipe (meta.yml) as needed. For details, see Writing the meta.yml 5. Generate the SHA256 key for your source code archive, as described in the example recipe using the openssl tool. As an alternative you can also go to the package description on PyPi from which you can directly copy the SHA Be sure to fill in the tests section. The simplest test will simply test that the module can be imported, as described in the example. 7. Remove all irrelevant comments in the meta.yaml file Checklist Ensure that the license and license family descriptors (optional) have the right case and that the license is correct. Note that case sensitive inputs are required (e.g. Apache 2.0 rather than APACHE 2.0). Ensure that you have included a license file if your license requires one most do. (see here) In case your project has tests included, you need to decide if these tests should be executed while building the conda-forge feedstock. Make sure that all tests pass sucessfully at least on your development machine. Recommended: run conda-build on your source code to ensure the recipe works locally. 3.3 What happens after the PR to staged-recipes is merged After the PR is merged, travis-ci will create a new git repo automatically. For example, the recipe for a package named pydstool will be moved to a new repository CI services (travis-ci, circleci, appveyor) will be enabled automatically and a build will be triggered automatically which will build the conda package and upload to If this is your first contribution, you will be added to the conda-forge team and given access to the CI services so that you can stop and restart builds. You will also be given commit rights to the new git repository. If you want to make a change to the recipe, send a PR to the git repository from a fork. Branches of the main repository are used for maintaining different versions only. 3.4 Avoid Dependencies Outside of Conda-Forge Do all of my package s dependencies have to be in conda(-forge) already? Short answer: yes. Long answer: In principle, as long as your dependencies are in at least one of your user s conda channels they will be able to install your package. In practice, that is difficult to manage, and we strive to have all dependencies built in conda-forge. Building all of the dependencies in conda-forge allow us greater assurance of ABI compatibility for the conda-forge packages. Only in extreme cases should you rely on a dependency outside of conda-forge. 10 Chapter 3. Creating Conda Recipes

15 If your dependencies do not change with the python version, or with the platform, consider making your build noarch, this will allow the recipe to build faster, and free some CI resources for other projects. 3.5 Optional: bld.bat and/or build.sh In many cases, bld.bat and/or build.sh files are not required. Pure Python packages almost never need them. If the build can be executed with one line, you may put this line in the script entry of the build section of the meta.yaml file with: script: "{{ PYTHON }} -m pip install. -vvv". Remember to always add pip to the host requirements. 3.6 Maintainer Role The maintainer s job is to: Keep the feedstock updated by merging eventual maintenance PRs from conda-forge s bots. Keep the feedstock on par with new releases of the source package by - Bumping the version number and checksum. - Making sure that feedstock s install and run requirements stay accurate. - Make sure the test requirements match those of the of the updated package. Answer eventual question about the package on the feedstock issue tracker. 3.7 Other Recipes in staged-recipes When a PR of a recipe to the staged-recipes repo is ready to go, it is merged into master. This will trigger a CI build specially designed to convert the recipe. However, for any number of reasons, the recipe may not be converted right away. In the interim, the recipe will remain in master until they can be converted. There is no action required on the part of recipe contributors to resolve this. It should have no impact on any other PRs being proposed. If any recipes pending conversion do cause issues for your submission, please for help. 3.8 Activate scripts Recipes are allowed to have activate scripts, which will be sourcedd or called as appropriate. It is generally recommended to avoid using activate scripts when another option is possible because people don t always activate environments the expected way and these packages may then misbehave. However when there is no other option, these are a reasonable option to use. When using them in a recipe, feel free to name them activate.bat, activate. sh, deactivate.bat, and deactivate.sh in the recipe. The installed scripts are recommended to be prefixed by the package name and a separating -. Below is some sample code for Unix and Windows that will make this install process easier. Please feel free to lift it. In build.sh: # Copy the [de]activate scripts to $PREFIX/etc/conda/[de]activate.d. # This will allow them to be run on environment activation. for CHANGE in "activate" "deactivate" do mkdir -p "${PREFIX}/etc/conda/${CHANGE}.d" (continues on next page) 3.5. Optional: bld.bat and/or build.sh 11

16 (continued from previous page) cp "${RECIPE_DIR}/${CHANGE}.sh" "${PREFIX}/etc/conda/${CHANGE}.d/${PKG_NAME}_$ {CHANGE}.sh" done In build.bat: setlocal EnableDelayedExpansion :: Copy the [de]activate scripts to %PREFIX%\etc\conda\[de]activate.d. :: This will allow them to be run on environment activation. for %%F in (activate deactivate) DO ( if not exist %PREFIX%\etc\conda\%%F.d mkdir %PREFIX%\etc\conda\%%F.d copy %RECIPE_DIR%\%%F.bat %PREFIX%\etc\conda\%%F.d\%PKG_NAME%_%%F.bat ) 12 Chapter 3. Creating Conda Recipes

17 CHAPTER 4 Writing the meta.yaml This document presents conda-forge rules, guidelines, and justifications about writing the meta.yaml file. 4.1 Build from Tarballs, Not Repos Conda-forge requires that building from tarballs using the url and fn keys in the build section. A recipe should not use the git_url, git_ver, and similar keys. There are three main reasons for this: Downloading the whole repo when you only need a single snapshot wastes the precious, constrained, and shared CI time and bandwidth. Repositories are not checksummed. Thus, using a tarball has a stronger guarantee that the download that is obtained to build from is in fact the intended package. On some systems (such as Windows), it is possible to not have permissions to remove a repo once it is created. This can be avoided by using a tarball. If a package does not have the ability to build from a tarball, this is considered a bug and should be reported upstream. In the worst case, the source can be patched to include the relevant build information. 4.2 Packaging the License Manually Certain software licenses, such as those in the GPL and Apache families, require that the text of the license be distributed with the package. This means that the about/license_file entry must be included in the meta.yaml. Unfortunately, the license isn t always included in the tarball of the source code. To get around this, the licence should be put in the recipe directory. It can then be refered to in the meta.yaml via, about: license_file: '{{ environ["recipe_dir"] }}/LICENSE' 13

18 4.3 Populating the hash Field If your package is on PyPi, you can get the sha256 hash from your package s page on PyPI; look for the SHA256 link next to the download link on your package s files page, e.g. <your-project>/#files. You can also generate a hash from the command line on Linux (and Mac if you install the necessary tools below). If you go this route, the sha256 hash is preferable to the md5 hash. To generate the sha256 hash: openssl sha256 your_sdist.tar.gz You may need the openssl package, available on conda-forge conda install openssl -c conda-forge. 4.4 Excluding a Platform Use the skip key in the build section along with a selector: build: skip: true # [win] A full description of selectors is in the conda docs. 4.5 Pinning packages Conda-smithy switches to conda-build=3. In conda-smithy 3.0.0, we use a central configuration file from conda-forge-pinning. for the build matrices and versions of specific packages. When a rerendering happens, conda-smithy will render the recipe using conda-build and output configuration files for each job and save them in a yaml file in.ci_support folder. For example there s a output configuration file for each OS, each python version, etc. These output configuration files are stripped to options that are used in the build and therefore a change in the config files in.ci_support folder implies that there needs to be a new build. Pinning of packages are handled by the same configuration file and conda-build. This means that packages need not be pinned manually. For eg: requirements: host: - gmp 6.1.* run: - gmp 6.1.* can be replaced by requirements: host: - gmp run: - gmp When there s a new ABI version of gmp (say 7.0), then conda-forge-pinning will be updated. A rerendering of a package using gmp will change. Therefore to check that a recipe needs to be rebuilt for updated pinnings, you only need to check if the package needs a rerender. 14 Chapter 4. Writing the meta.yaml

19 Note that numpy is an exception to this. See Building Against NumPy section. If a package is not in conda-forge-pinning, then the pinning needs to be done manually. If the package is a C/C++ library with a C/C++ API that is consumed and linked to by other libraries, then that package is a candidate to be added to conda-forge-pinning. Please open an issue in conda-forge-pinning-feedstock for discussion. If the constraints in conda-forge-pinning are not strict enough, you can override them by changing back to pinning the package with a version manually. You can make a pinning stricter by adding {{ pin_compatible('gmp', max_pin='x.x.x') }} to run requirements. If you need to remove a pinning in rare cases like linking the package statically or if the package is used with dlopen instead of linking, then you can do, build: ignore_run_exports: - gmp There is additional documentation on this pinning scheme in the conda docs. 4.6 Using conda-build s run_exports feature Conda-build has a feature called run_exports that allows recipe builders to specify what downstream consumers of their package need as runtime dependencies when a given package is used as a build-time (host) dependency. There s documentation on run_exports in conda-build s documentation. For conda-forge specifically, we encourage the use of run_exports, as it places the burden of understanding binary compatibilty bounds on the upstream package maintainers, rather than the downstream package consumers. We feel that the upstream maintainers are more likely to understand the compatibility bounds of their package. Downstream maintainers are still free to override any run_exports entries, using the build/ignore_run_exports key, as documented in the conda-build documentation. When should you think about using run_exports for your package? Does your package produce a shared library that other people will link against? Does the version of your package used at build time determine which version of your package is required at run time? If these are true, you should use run_exports. You can refer to your package with the pin_subpackage jinja2 function, again described in conda-build s documentation. It is not always completely clear how a given package is going to be used. For example, numpy may be used either as a python package, and it also has a C library that can be linked against. The former usage would not require run_exports, but the latter would. In this scenario, it may be advantageous to split the package into distinct metapackages that may share a common parent containing the actual files, but with each metapackage defining different pinning behavior. Anaconda does this for numpy. You can see the recipe at master/recipe/meta.yaml - the general idea is that the numpy-devel package should be used when a package is building against the C interface (i.e. it needs the compatibility bound), and the numpy package should be used when a package is using only the python interface. In general, it is not necessary to split up packages. At conda-forge, we only advise it when it greatly reduces package size, or when it helps remove dependencies that would otherwise be unnecessarily included. 4.7 Build matrices Currently, python, vc, r-base will create a matrix of jobs for each supported version. If python is only a build dependency and not a runtime dependency (eg: build script of the package is written in Python, but the package is not dependent on python), use build section 4.6. Using conda-build s run_exports feature 15

20 Following implies that python is only a build dependency and no Python matrix will be created. build: - python host: - some_other_package Note that host should be non-empty or compiler jinja syntax used or build/merge_build_host set to True for the build section to be treated as different from host. Following implies that python is a runtime dependency and a Python matrix for each supported python version will be created. host: - python conda-forge.yml s build matrices is removed in conda-smithy=3. To get a build matrix, create a conda_build_config.yaml file inside recipe folder. For example following will give you 2 builds and you can use the selector vtk_with_osmesa in the meta.yaml vtk_with_osmesa: - False - True You need to rerender the feedstock after this change. 4.8 Compilers conda-build=3 gives the ability to use Anaconda 5 compilers, conda-forge is not using these compilers yet. Reason for holding out on using the Anaconda 5 compilers is that the packages built by them are sometimes incompatible with the packages built with the older compilers in CI platforms conda-forge has been using. All the dependencies of a package should be compiled (This is not true for all cases, but it s better to be cautious) with the new compilers before using the new compiler in a package. This presents a problem in that rebuilding a package will break the dependent packages. Therefore, conda-forge has decided to rebuild all of the packages and upload them all at once. More details on how this is done will be communicated in the future. However, using the {{ compiler('cxx') }} is supported in conda-forge, but it installs the toolchain package which activates the compilers in the CI environment. If you were using toolchain or gcc build deps, consider using the following, requirements: build: - {{ compiler('c') }} - {{ compiler('cxx') }} - {{ compiler('fortran') }} Note that appropriate compiler runtime packages will be automatically added to the package s runtime requirements and therefore there s no need to specify libgcc or libgfortran. There is additional information about how conda-build 3 treats compilers in the conda docs. 4.9 Building Against NumPy If you have a package which links* against numpy you can build against the oldest possible version of numpy that is forwards compatible. With conda-build 3, we can leave the pin empty for build-time, and conda-build will use the 16 Chapter 4. Writing the meta.yaml

21 numpy key from conda_build_config.yaml. We can also utilize conda-build s dynamic pinning with its pin_compatible function to evaluate the numpy pin based on the version that actually gets used at build time. If you don t want to make things complicated you can use host: - numpy run: - {{ pin_compatible('numpy') }} At the time of writing, above is equivalent to the following, host: - numpy # [unix] - numpy # [win] run: - numpy >=1.9.3,<2.0.a0 # [unix] - numpy >=1.11.3,<2.0.a0 # [win] * In order to know if your package links against numpy check for things like numpy.get_include() in your setup.py, or if the package uses cimport. Notes 1. you still need to respect minimum supported version of numpy for the package! That means you cannot use numpy 1.9 if the project requires at least numpy 1.12, adjust the minimum version accordingly! host: - numpy 1.12.* run: - {{ pin_compatible('numpy') }} At the time of writing, above is equivalent to the following, host: - numpy run: - numpy >=1.12.3,<2.0.a0 2. if your package supports numpy 1.7, and you are brave enough :-), there are numpy packages for 1.7 available for Python 3.4 and 2.7 in the channel. Deprecated Adding numpy x.x to the host and run sections translates to a matrix pinned to all available numpy versions (e.g. 1.11, 1.12 and 1.13). In order to optimize CI ressources usage this option is now deprecated in favour of the apporach described above Building noarch packages The noarch: python directive, in the build section, makes pure-python packages that only need to be built once. This drastically reduces the CI usage, since it s only built once (on CircleCI), making your build much faster and freeing resources for other packages Building noarch packages 17

22 noarch: python can be used to build pure Python packages that: Do not perform any Python version specific code translation at install time (i.e. 2to3). Have fixed requirements; that is to say no conditional dependencies depending on the Python version, or the platform ran. (If you have for example backports # [py27]) in the run section of meta.yml, your package can t be noarch, yet.) Do not use selectors to skip building the recipe on a specific platform or for a specific version of python (e.g. skip: True # [py2k]). Note that while noarch: python does not work with selectors, it does work with version constraints, so skip: True # [py2k] can sometimes be replaced with a constrained python version in the build/run subsections: say python >=3 instead of just python. noarch: python can also work with recipes that would work on a given platform except that we don t have one of its dependencies available. If this is the case, when the install runs conda will fail with an error message describing which dependency couldn t be found. If the dependency is later made available, your package will start working without having to change anything. Note though that since noarch packages are built on Linux, currently the package must be installable on Linux. To use it, just add noarch: build: noarch: python python in the build section like, If you re submitting a new recipe to staged-recipes, that s all you need. In an existing feedstock, you ll also need to re-render the feedstock, or you can just ask the webservice to add it for you and rerender: please add noarch: python in an open PR Build Number The build number is used when the source code for the package has not changed but you need to make a new build. For example, if one of the dependencies of the package was not properly specified the first time you build a package, then when you fix the dependency and rebuild the package you should increase the build number. When the package version changes you should reset the build number to Use pip Normally Python packages should use this line: build: script: "{{ PYTHON }} -m pip install. --no-deps --ignore-installed --no-cache-dir - vvv" as the installation script in the meta.yml file or bld.bat/build.sh script files, while adding pip to the host requirements: requirements: host: - pip These options should be used to ensure a clean installation of the package without its dependencies. This helps make sure that we re only including this package, and not accidentally bringing any dependencies along into the conda package. 18 Chapter 4. Writing the meta.yaml

23 Note that the --no-deps line means that for pure-python packages, usually only python and pip are needed as build or host requirements; the real package dependencies are only run requirements Downloading extra sources and data files conda-build 3 supports multiple sources per recipe. Examples are available in the conda-build docs Downloading extra sources and data files 19

24 20 Chapter 4. Writing the meta.yaml

25 CHAPTER 5 BLAS & Numpy & Friends BLAS metapackage Version will have two values X.Y * X represents changes to the metapackage. * Y represents priority of BLAS (if we change priorities BLASes X must be bumped, if we want to prioritize something new over OpenBLAS we do not need to change X) 1 is OpenBLAS 0 is None (no BLAS whatsoever) needs to have version stay the same across all variants. build number cannot be touched (it won t be in the string anyways) no pinning of BLAS inside the metapackages (dependencies can pin this) to preserve a BLAS in an environment it is recommend to add pinned to conda-meta and specify down to the build string what is the expected BLAS To install a specific variant, conda install blas=1.0=none / conda install blas=1. 0=openblas. It is hoped the syntax will be improved in conda. * In the future, with some fixes to conda will allow for syntax like conda install blas=*=openblas. We should keep an eye on this. (maybe even conda install blas:openblas) There will be two variants initially: * openblas Numpy package * noblas - no BLAS optimisations (e.g. for reasons of smaller installations) version + build number must always be greater than or equal to that in defaults. If not, defaults numpy will be chosen, complete with mkl 21

26 * to make this simple we can pick a high build number so this is prioritized 100 and then bump from there Should make the build number ranges tied to BLAS X version above. Build number should start at (X+1)*100. Means OpenBLAS starts at 200. No BLAS starts at 100. Unfortunately the release breaks this rule so we will have No BLAS at 101. * if defaults gains a newer version and build without conda-forge updating, users will be prompted to upgrade to the defaults numpy. Even if a user does this, as soon as an equivalent build is available on conda-forge, they will be prompted to update to their previous variant will track the blas_{{ variant }} feature enabled by the BLAS metapackage will pin the specific blas package versions (e.g. openblas.) SciPy, scikit-learn, etc. package Same thing as NumPy Add numpy dependency as if linking occurs 22 Chapter 5. BLAS & Numpy & Friends

27 CHAPTER 6 Building on Windows This document presents conda-forge and conda-build information and examples when building on Windows. 6.1 Simple CMake-Based bld.bat Some projects provide hooks for CMake to build the project. The following example bld.bat file demonstrates how to build a traditional, out-of-core build for such projects. CMake-based bld.bat: setlocal EnableDelayedExpansion :: Make a build folder and change to it. mkdir build cd build :: Configure using the CMakeFiles cmake -G "NMake Makefiles" ^ -DCMAKE_INSTALL_PREFIX:PATH="%LIBRARY_PREFIX%" ^ -DCMAKE_PREFIX_PATH:PATH="%LIBRARY_PREFIX%" ^ -DCMAKE_BUILD_TYPE:STRING=Release ^.. if errorlevel 1 exit 1 :: Build! nmake if errorlevel 1 exit 1 :: Install! nmake install if errorlevel 1 exit 1 The following feedstocks are examples of this build structure deployed: 23

28 libping pugixml 6.2 Building for different VC versions On Windows, different Visual C versions have different ABI and therefore a package needs to be built for different Visual C versions. Packages are tied to the VC version that they were built with and some packages have specific requirements of the VC version. For example, python 2.7 requires vc 9 and python 3.5 requires vc 14. With conda-build 3.x, vc can be used as a selector when using the compiler jinja syntax. requirements: build: - {{ compiler('cxx') }} To skip building with a particular vc version, add a skip statement. build: skip: true # [vc<14] requirements: build: - {{ compiler('cxx') }} 24 Chapter 6. Building on Windows

29 CHAPTER 7 Testing in Recipes All recipes need tests. Here are some tips, tricks, and justifications. How you shold test depends on the type of package (python, c-lib, command-line tool,... ), and what tests are available for that package. But every conda pacakge must have at least some tests. 7.1 Simple Existence Tests Sometime tests for the underlying code base don t exist. And sometimes test suites simply take too long to run on limited CI infrastructure. Or sometimes they take too much bandwidth. Whatever the reason, conda-forge may not always be able to execute the prescribed test suite. However, this is no reason for the recipe to not have tests. At the very least we want to verify that the package has installed the desired files in the desired locations. This is called existence testing. Existence testing can be accomplished in the meta.yaml file in the test/commands block. On posix systems, use the test utility and the $PREFIX variable. On Windows, use the exist command. See below for an example Simple Existence Testing Example test: commands: - test -f $PREFIX/lib/libboost_log$SHLIB_EXT # [unix] - if not exist %LIBRARY_LIB%\\boost_log-vc140-mt.lib exit 1 # [win] 7.2 Testing Python Packages For the best information about testing, see the conda build docs test section. 25

30 7.2.1 Testing Importing The minimal test of a python package should make sure that the package can be successfully imported. This can be accomplished with this stanza in the meta.yaml: test: imports: - package_name Note that package_name is the name imported by python; not necessarily the name of the conda package (they are sometimes different). Testing for an import will catch the bulk of the packaging errors, generally including presence of dependencies. However, it does not assure that the package works correctly. In particular, it doesn t test if if it works correctly with the versions of dependencies used. It is good to run some other tests of the code itself (the test suite) if possible Running Unit Tests The trick here is that there are multiple ways to run unit tests in Python, including nose, pytest, etc. Also, some packages install the tests with the package, and thus they can be run in place, while others keep the tests with the source code, and thus can not be run straight from an installed package Test Requirements Sometimes there are packages required to run the tests that are not required to simply use the package. This is usually a test-running framework, such as nose or pytest. You can ensure that it is included by adding it to requirements in the the test stanza: test: imports: - package_name... requires: - pytest Built-in Tests Some packages have testing built-in. In this case, you can put a test command directly in the test stanza: test:... commands: python -c "import package_name; package_name.tests.runall()" Alternatively, you can add a file called run_test.py in the recipe that will be run at test time. This allows an arbitrarily complicated test script pytest Tests If the tests are installed with the package, pytest can find and run them for you with the following command: 26 Chapter 7. Testing in Recipes

31 test: requires: - pytest commands: - py.test --pyargs package_name Command Line Utilities If a python package installs command line utilities, you probably want to test that they were properly installed: test: commands: - util_1 --help If the utility actually has a test mode, great. Otherwise simply invoking --help or --version or something will at least test that it is installed and can run Tests Outside of The Package Note that conda-build runs the tests in an isolated environment after installing the package thus, at this point it does not have access to the original source tarball. This is to ensure that the test environment is as close as possible to what an end-user will see. This makes it very hard to run tests that are not installed with the package. 7.3 Docker Tests Run Docker Tests Locally for Feedstock If you want to run the docker tests for a feedstock locally, go to the root feedstock directory and run, the.circleci/ run_docker_build.sh script (or ci_support/run_docker_build.sh in older feedstocks). One should also specify the environment variable CONFIG, to select one of the *.yaml config files in.ci_support to use for the build (this is not needed for older feedstocks). $ cd my-feedstock $ CONFIG="linux_"./.circleci/run_docker_build.sh Run Docker Tests Locally for Staged Recipes If you want to run the docker tests for the staged-recipes repository locally, go to the root repository directory and run the.circleci/run_docker_build.sh script. $ cd staged-recipes $./.circleci/run_docker_build.sh 7.3. Docker Tests 27

32 7.4 Should a Recipe Run All of a Package s Tests? No. A recipe does not have to run all of a package s unit or integration tests. Sometimes, this might even be impossible due to timeouts on CI services. Just test sufficiently to know that the package is in good working order. 28 Chapter 7. Testing in Recipes

33 CHAPTER 8 CI Services Here we describe common issues with the CI Services that conda-forge builds. 8.1 Travis CI (OS X) Travis CI is used to build packages for OS X. After merging a staged-recipes pull request, it might be necessary to force sync your repositories in Travis CI to see the reload and cancel buttons. To do this please visit org/profile and click Sync accounts. 8.2 CircleCI (Linux, OSX) Circle CI is a container-based CI service that conda-forge uses to build linux packages. It can optionally build OSX packages Using Circle for both Linux and OSX To use CircleCI for OSX, add the following to conda-forge.yml in the root of the feedstock. provider: osx: circle CircleCI for OSX should be used for OSX only when Travis-CI resources (50 minutes of build time per job) is not enough as CircleCI gives more resources (2 hours of build time per job). Note that you need to rerender the feedstock once this change has been made. 29

34 8.2.2 Enabling Circle on your Fork If for some reason Circle CI is not triggering build from forks, Circle can be manually added for each fork. Circle calls this Adding a Project and the official Circle s documentation is available here. This effectively amount to going to the Add Projects page, finding the fork that you wish to enable, and clicking the Build Project button. This is not normally needed. If CircleCI lacks permissions to checkout the source code, it will produce an error like follows: Cloning into '.'... Warning: Permanently added the RSA host key for IP address ' ' to the list of known hosts. Permission denied (publickey). fatal: Could not read from remote repository. When this happens for a feedstock, it can be fixed using the webservice, by posting the following please update circle Otherwise (e.g. in a PR to staged-recipes), here are some things you can try: Log in and out of Circle CI. Revoke Circle CI s access and then enable it again. In the Checkout SSH keys section of your Circle CI project settings, press add user key. 30 Chapter 8. CI Services

35 CHAPTER 9 conda-smithy rerender conda-forge uses conda-smithy to re-render the feedstock and update the feedstock configuration. This action updates the README, CI configuration, the build matrix, fix problems with the feedstock configuration, etc. The bot issues re-rendering PRs every now and then, but sometimes the users may need to issue them manually instead waiting for the bot. Here is how, and when, to re-render. 9.1 How to re-render? There are 2 options, 1. Using the webservice to run conda-smithy on the cloud. See Automatic services 2. Run conda-smithy locally on your machine. To do this, First step is to install conda-smithy in your root environment conda install -c conda-forge conda-smithy now, from the root directory of the feedstock, type: conda smithy rerender [--commit {edit auto}] and commit the changes! Optionally one can commit the changes automatically. To do this use the --commit/-c option. By default, condasmithy will have git open an editor to write the commit message. It will also provide a default commit message and show the changes to be added. If you wish to do this automatically, please just use --commit auto/-c auto and it will commit with the stock re-rendering message. Note that you may get a warning message regarding the tokens. It is safe to ignore that as they are not needed for re-rendering. 31

36 9.2 When should we re-render? We need to re-render when there are changes the following parts of the feedstock: the yum_requirements.txt; the platform configuration (skip sections); build issues that a feedstock configuration update will fix (follow us on gitter to know about those); updates in the build matrix due to new versions of Python, NumPy, PERL, R, etc. 32 Chapter 9. conda-smithy rerender

37 CHAPTER 10 conda-forge gotchas 10.1 Using multiple channels It is quite common to install a package from conda-forge and, when trying to use it, see an error like (OS X example): ImportError: dlopen(.../site-packages/rpy2/rinterface/_rinterface.so, 2): Library not Referenced from:.../site-packages/rpy2/rinterface/_rinterface.so Reason: image not found That happens because either the correct version of icu, or any other package in the error, is not present or the package is missing altogether. Once can confirm by issuing the command conda list and searching for the package in question Why that happens? The conda-forge and defaults are not 100% compatible. In the example above it is known that defaults uses icu 54.* while conda-forge relies on icu 56.*, that mismatch can lead to errors when the install environment is mixing packages from multiple channels. Note: All of conda-forge software pinning can be found at: conda-forge-pinning-feedstock/blob/master/recipe/conda_build_config.yaml How to fix it? Newer conda versions introduced a channel priority feature. See for more information. One possible solution is to add the conda-forge channel on top of defaults in your condarc file when using conda-forge packages. This will ensuring that all the dependencies will come from the conda-forge channel. Here is how a.condarc file would look like: 33

38 $ cat.condarc channels: - conda-forge - defaults In addition to the channel priority we recommend to always install your packages inside a new environment instead the root environment from anaconda/miniconda. Using envs make it easier to debug problems with packages and ensure the stability of your root env Using a fork vs a branch when updating a recipe All maintainers are given push access to the feedstocks that they maintain. This means that a maintainer can create branches in the main repo. For updates, using a branch in the main repo is discouraged because, 1. CI is run on both the branch and the PR. This wastes Appveyor and Travis-CI resources. 2. Branches are automatically published. This means if you push a version update to a branch and then create a PR, conda packages will be published to anaconda.org before the PR is merged. For these reasons maintainers are asked to fork the feedstock, push to a branch in the fork and then open a PR to the conda-forge repo. Branches in the main repo are used for, 1. Maintaining a LTS branch of a package. For eg. master branch of python-feedstock builds 3.6.x, while 3.5 branch builds 3.5.x versions of python. 34 Chapter 10. conda-forge gotchas

39 CHAPTER 11 Automatic services conda-forge is running a webservice on heroku called conda-forge-webservices. The following services are run by default on a feedstock: It will lint the recipes in the PRs and report back whether recipe is in excellent condition or not. When maintainers are added to a recipe, the maintainer will be added to the team and given push access. The webservice also listens to issue and PR comments so that you can ask for the following services to be done. please rerender Entering the above phrase in a PR of a feedstock will rerender the feedstock and push the changes to your PR. Make sure to tick the Allow edits from maintainers. button locate at the bottom of the right side bar of the PR. If you say this phrase in an issue comment, the bot will create a new pull request with the requested re-rendering completed. please add noarch: python Entering the above phrase in a PR or issue of a feedstock will add noarch: feedstock for you. python to the build and rerender the please update for conda-build 3 This command will attempt to update a recipe to the new conda-build 3 format. It can be sent either in an issue or a PR. Note that this update command is kind of a hack, and things might go wrong. Make sure to look over the changes, and ask for help if you re not sure about something. 35