Contributing Guide

Attention

A GitHub account is required to begin contributing to AstroConda

Guidelines

Attention

The following packaging guidelines are subject to change at any time.

  • Please be respectful when commenting on pull-requests or issues.
  • If your contribution is not accepted into AstroConda, as a general courtesy, you will be given a clear and concise reason.
  • As a contributor you may not claim exclusive rights to a particular recipe.
  • You are free to maintain a recipe in AstroConda by issuing regular pull requests.
  • Everyone is welcome to improve upon recipes so long as the changes do not introduce packaging conflicts.
  • Abandoned recipes may be moved into the deprecated directory at any time without warning. (i.e. The package no longer compiles, has been obsoleted, or presents a conflict that cannot be resolved, etc).
  • Packages derived from deprecated recipes will remain available in AstroConda for historical purposes (i.e. to preserve backwards compatibility).

Bugs, questions, and requests

Please open a new issue or send us a pull request for bugs, feedback, questions, or enhancements.

Adding a recipe to astroconda-contrib

In this example we will be adding a new recipe to the AstroConda repository for sympy, the symbolic mathematics library.

Navigate to the astroconda-contrib repository on GitHub, login, and create a fork (or click here to have your fork created automatically).

Now that you have a fork of astroconda-contrib, go ahead and clone it to your system:

git clone https://github.com/[Your_Account]/astroconda-contrib
cd astroconda-contrib

To get started adding our recipe, create a new branch and name it sympy-contrib:

git checkout -t -b sympy-contrib

Git will automatically switch your branch from master to sympy-contrib as denoted by the following output:

Branch sympy-contrib set up to track local branch master.
Switched to a new branch 'sympy-contrib'

If you have taken the liberty of looking around the astroconda-contrib directory, you will have noticed a bunch of directories are sitting in there all named by-package. So let’s keep things simple and straight forward. Go ahead and create a directory and name it sympy, and proceed inside:

mkdir sympy
cd sympy

Note

This is not an full Conda packaging tutorial. For more information about creating recipes from scratch, please refer to the conda-build documentation.

Hint: Investigate the contents of the recipes in astroconda-contrib. For most cases, copying an existing recipe and changing its values will suffice.

Copy the contents of the astroconda-contrib/template recipe. Three files bld.bat, build.sh, and meta.yaml will be copied to your working directory:

cp ../template/* .

Go ahead and open meta.yaml with your favorite plain-text editor:

Caution

It is highly recommended that you enable “tabs to spaces” for your editor. Tab widths are unpredictable and may cause Conda’s YAML parser to fail.

vim meta.yaml

The general structure of the file is as follows:

# These directives are commented here due to Pygments
# failing to parse this section of code.
# ... They are not commented in the real file.

#{% set name = '' %}
#{% set version = '' %}
#{% set number = '0' %}

about:
    # Package homepage
    home:
    # Package license
    license:
    # A brief description
    summary:

package:
    # Define these values above
    name: {{ name }}
    version: {{ version }}

build:
    # Define this value above
    number: {{ number }}

source:
    fn: {{ name }}-{{ version }}.tar.gz
    url: http://example.com/example/{{ name }}-{{ version }}.tar.gz

requirements:
    build:
    # Dependencies required at BUILD-TIME go here
    - setuptools
    - python x.x
    run:
    # Dependencies required at RUN-TIME go here
    # - ...

#test:
#   imports:
#   # - (e.g. a_python_module)
#
#   commands:
#   # - (e.g. program --help)

First, let’s fill in the blanks. Modify the JINJA2 template markers for name, version:

{% set name = 'sympy' %}
{% set version = '1.0' %}

Fill in the about section with relevant information regarding the package:

about:
    home: http://sympy.org
    license: GPL
    summary: Python library for symbolic mathematics

Next, modify the source section’s url so that it points to sympy’s source archive (on the internet):

source:
    fn: {{ name }}-{{ version }}.tar.gz
    url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz

What’s with the never-ending stream of bracket encapsulated keywords, you ask? Conda uses JINJA2, a basic template system that provides basic string interpolation within recipes. This comes in handy if, let’s say, you decide to build a more recent version of sympy, you need only modify the first two variable definitions in this file (assuming the URL structure has not changed).

The requirements section may be confusing to some, so let’s clarify the distinction between build and run before diving in. The build section defines Conda packages required at compile-time (i.e. python setup.py install), whereas the run section lists Conda packages required at install-time (i.e. conda install [package]).

As recipe maintainer the method used to dependency discover is that of trial and error. For many Python packages obtained via PyPi, it is easy enough to visually examine setup.py or requirements.txt to get a good idea of the recipes you need to depend on. Some package may require several iterations of executing conda build and returning to your recipe in the editor to append more packages.

As we can see below, sympy requires mpmath, setuptools and python to build the recipe, but only mpmath and python to run it successfully after installation:

requirements:
    build:
    - mpmath
    - setuptools
    - python x.x
    run:
    - mpmath
    - python x.x

What does the x.x imply exactly? This instructs conda build not to proceed unless python=[version] has been issued as an argument on the command-line. If x.x is omitted here, the recipe will choose the version of Python currently active in your environment. In most cases it is best to be explicit rather than implicit when it comes to defining version requirements in Conda.

The test section defines few useful lists, imports, commands, and requires. While these are not required to be used in any given recipe, we do use them in AstroConda. The imports section is a list of Python module imports, the commands are executed in a basic shell environment, and finally requires defines any extraneous packages to be installed into the environment before running the tests.

test:
    imports:
        - sympy

    #commands:
    #   - no shell commands to execute

    #requires:
    #   - does not require any extra testing-related packages

If sympy provided a command-line utility named sympy-show, you would use the commands section to verify the utility’s functionality. A simple example of this would be to output a usage statement.

test:
    # ...
    commands:
        - sympy-show --help

If a program returns greater than zero conda build will fail as if it observed an error. Not all programs return zero after issuing --help (or an equivalent argument). Due to this, you may need to omit this style of test.

It will not hurt to keep the commands section populated but disabled with a note describing why it doesn’t work. Others will find this information useful. Given this scenario, the optimal approach would be to contact the developers and plead with them to normalize the exit value.

Below is our sympy final recipe. Despite the overwhelming use of JINGA2 in our example, things are looking pretty streamlined.

{% set name = 'sympy' %}
{% set version = '1.0' %}
{% set number = '0' %}

about:
    home: http://sympy.org
    license: GPL
    summary: Python library for symbolic mathematics

source:
    fn: {{ name }}-{{ version }}.tar.gz
    url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz

requirements:
    build:
    - mpmath
    - setuptools
    - python x.x
    run:
    - mpmath
    - python x.x

test:
    imports:
        - sympy

The template directory copied earlier in this tutorial contains two basic python build scripts for both *Nix (build.sh) and Windows (bld.bat), and is coincidentally compatible with the example we’re using here. Not all Python packages (especially Makefile-based packages) will compile successfully using this “one-liner” template. Always refer to the INSTALL file or equivalent documentation for the program you are attempting to compile to learn more about what the package expects from the end-user at compile-time.

Before we can issue a pull request on GitHub, we first ensure it builds, tests, and installs properly on our local system. To do this we need to change our directory to one level above the recipe.

cd ..
# i.e. /path/to/astroconda-contrib

Now run conda build to compile our sympy recipe into a Conda package. In the example below we are building against Python 3.5:

conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=3.5 sympy

That’s probably a bit more involved than you thought. Let’s break it down. We issue -c [URL] which instructs the build to utilize the AstroConda channel while checking for package dependencies (i.e. the recipe’s requirements section). Secondly, we issue --skip-existing to prevent conda build from rebuilding dependencies discovered in the local astroconda-contrib directory. That is to say, if a package defined as a requirement exists remotely, it will then download and install it, rather than rebuild it from scratch. --python= is self-explanatory, and the final argument is the name of the recipe(s) we intend to build.

At this point, if the build was successful, our Conda package (a bzipped tarball) called sympy-1.0-py35_0.tar.bz2 is emitted to /path/to/miniconda3/conda-bld/[os-arch]/. This directory is a local Conda package repository.

To install this new sympy package and interact with it ourselves you could run the following:

conda create -n sympy_test --use-local sympy
source activate sympy_test

Then manually verify the package is working:

python

And checking it out for yourself:

>>> import sympy
>>> sympy.__file__
'/path/to/miniconda3/envs/sympy_test/lib/python3.5/site-packages/sympy/__init__.py'

Now that you have verified the recipe is fully functional and are happy with the outcome, it’s time to create a pull request against astroconda-contrib main repository.

Push your sympy-contrib branch up to your fork on GitHub:

git push origin sympy-contrib

It is recommended that you familiarize yourself with GitHub pull requests before proceeding (see: tutorial).

Using GitHub, you will want to browse to your astroconda-contrib fork, select the sympy-contrib branch from the drop-down menu (the default will read: “Branch: master”, next to a black downward-pointing caret). Once selected, click the large green button labeled: “New pull request”.

From here, you may wish to edit the title of your pull request and add initial comments or notes regarding what you have done, or list any work that may still need to be done. After submitting your pull request, a member of the Science Software Branch at STScI, or fellow contributors will review the requested changes, ask questions, offer feedback or advice.

At this point if everything appears to be in order your recipe will likely be merged, built, and incorporated into AstroConda!

Updating a recipe in astroconda-contrib

Let’s assume time has passed and our sympy package from the previous example is no longer up to date with what’s generally available on GitHub. Updating recipes is a fairly straight forward process.

At the top of the file you will remember we have a few variables defined encapsulated by {% %}. These jinja2 variables control the name, version, and build revision of the recipe. Using variable interpolation saves time, because it’s much easier to edit a single variable that effects an entire recipe, than it is to search/replace specific fields. Typos are also much easier to spot.

{{ name }}, {{ version }} and {{ number }} expand to sympy, 1.0 and 0 respectively:

{% set name = 'sympy' %}
{% set version = '1.0' %}
{% set number = '0' %}

[...]

build:
    number: {{ number }}

[...]
source:
    fn: {{ name }}-{{ version }}.tar.gz
    url: https://github.com/{{ name }}/{{ name }}/releases/download/{{ name }}-{{ version }}/{{ name }}-{{ version }}.tar.gz

So to update sympy to version 1.2, for example, you would perform the following steps in your forked astroconda-contrib repository.

Checkout a new branch

git checkout -tb update-sympy master

Doing this ensures your new branch is based on master rather than your current branch, if any. It also keeps your master branch pristine, avoiding merge conflicts in the future.

Make your modifications

$EDITOR sympy/meta.yaml

[...]
{% set version = '1.2' %}
#                  ^ Was '1.0', but not anymore.

Now save and exit your editor.

Review your modifications

As stated earlier, the fastest way to find out whether your recipe works correctly is to try building it for yourself.

conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=2.7 sympy
conda build -c http://ssb.stsci.edu/astroconda --skip-existing --python=3.5 sympy

Did it work? If not, review the error message and make changes accordingly.

Commit your modifications

Assuming you are able to build the package locally, then you’re ready to push your changes up to your fork.

git add sympy/meta.yaml
git commit -m 'Update sympy 1.0 -> 1.2'
git push origin update-sympy

Open a pull request

See: Using Pull Requests

  1. Using your browser, visit the update-sympy branch of your fork: https://github.com/YOUR_ACCOUNT/astroconda-contrib/tree/update-sympy
  2. Click the gray “New pull request” button
  3. Fill out the pull request form
  4. Click the green “Create pull request” button

That’s all there is to it. One of our maintainers will review the pull request and get back to you.