Skip to main content

Artifacts

warning

This page is in active development, some content may be inaccurate.

conda environments can be created in a few different ways. conda-store creates "artifacts" (corresponding to different environment creation options) that can be shared with colleagues and can be used to reproduce environments. In the conda-store UI, these are available in the "Logs and Artifacts" section at the end of the environment page.

The following sections describe the various artifacts generated and how to create environments with them.

YAML file (pinned)​

YAML files that follow the conda specification is a common way to create environments. conda-store creates a "pinned" YAML, where all the exact versions of requested packages (including pip packages) as well as all their dependencies are specified, to ensure new environments created match the original environment as closely as possible.

A pinned YAML file is generated for each environment ta is built. This includes pinning of the `pip`` packages as well.

info

In rare cases, the completely pinned packages may not solve because packages are routinely marked as broken and removed.

conda-forge (default channel in conda-store) has a policy that packages are never removed but are marked as broken. Most other channels do not have such a policy.

Assuming you have conda installed, to create a conda environment (on any machine) using this file:

  1. Click on "Show yml file" link in the conda-store UI to open the file in a new browser tab.
  2. Save the file with: Right-click on the page -> Select "Save As" -> Give the file a meaningful name (like environment.yml)
  3. Run the following command and use the corresponding filename: 
    conda env create --file <environment.yml>

Lockfile​

A conda lockfile is a representation of only the conda dependencies in a given environment. conda-store created lockfiles using the conda-lock project.

warning

This file will not reproduce the pip dependencies in a given environment. It is usually a good practice to not mix pip and conda dependencies.

Click the lockfile icon to download the lockfile. First install conda-lock if it is not already installed.

conda install -c conda-forge lockfile

Install the locked environment file from conda-store.

conda-lock install <lockfile-filename>

conda-pack archive​

Conda-Pack is a package for creating tarballs of given Conda environments. Creating a Conda archive is not as simple as packing and unpacking a given directory. This is due to the base path for the environment that may change. Conda-Pack handles all of these issues. Click the archive button and download the given environment. The size of the archive will be less than the size seen on the environment UI element due to compression.

conda install -c conda-forge conda-pack

Install the Conda-Pack tarball. The directions are slightly complex. Note that my_env can be any name in any given prefix.

mkdir -p my_env
tar -xzf <conda-pack-tarfile>.tar.gz -C my_env

source my_env/bin/activate

conda-unpack

Docker images​

note

Docker image creation is currently only supported on Linux.

conda-store acts as a docker registry which allows for interesting ways to handle Conda environment. In addition this registry leverages conda-docker which builds docker images without docker allowing for advanced caching, reduced image sizes, and does not require elevated privileges. Click on the docker link this will copy a url to your clipboard. Note the beginning of the url for example localhost:8080/. This is required to tell docker where the docker registry is located. Otherwise by default it will try and user docker hub. Your url will likely be different.

The conda-store docker registry requires authentication via any username with password set to a token that is generated by visiting the user page to generate a token. Alternatively in the conda_store_config.py you can set c.AuthenticationBackend.predefined_tokens which have environment read permissions on the given docker images needed for pulling.

docker login -u token -p <conda-store-token>
docker pull <docker-url>
docker run -it <docker-url> python

General usage​

docker run -it localhost:8080/<namespace>/<environment-name>

If you want to use a specific build (say one that was built in the past and is not the current environment) you can visit the specific build that you want in the UI and copy its docker registry tag name. The tag name is a combination of <specification-sha256>-<build date>-<build id>-<environment name> that we will refer to as build key. An example would be localhost:5000/filesystem/python-numpy-env:583dd55140491c6b4cfa46e36c203e10280fe7e180190aa28c13f6fc35702f8f-20210825-180211-244815-3-python-numpy-env.

docker run -it localhost:8080/<namespace>/<environment-name>:<build_key>

On Demand Docker Image​

conda-store has an additional feature which allow for specifying the packages within the docker image name itself without requiring an actual environment to be created on the conda-store UI side.

The following convention is used <registry-url>:<registry-port>/conda-store-dynamic/. After conda-store-dynamic you specify packages needed separated by slashes. Additionally you may specify package constraints for example <=1.10 as .lt.1.10.

As full example support we want python less than 3.8 and NumPy greater than 1.0. This would be the following docker image name. <registry-url>:<registry-port>/conda-store-dynamic/python.lt.3.8/numpy.gt.1.0. conda-store will then create the following environment and the docker image will download upon the docker image being built.

Installers​

conda-store uses constructor to generate an installer for the current platform (where the server is running):

  • on Linux and macOS, it generates a .sh installer
  • on Windows, it generates a .exe installer using NSIS.

conda-store automatically adds conda and pip to the target environment because these are required for the installer to work.

Also note that constructor uses a separate dependency solver instead of utilizing the generated lockfile, so the package versions used by the installer might be different compared to the environment available in conda-store. There are plans to address this issue in the future.

Existing Deployments​

conda-store saves environment settings and doesn't automatically update them on startup (see CondaStore.ensure_settings). Existing deployments need to manually enable installer builds via the admin interface. This can be done by going to <CondaStoreServer.url_prefix>/admin/setting/<namespace>/<env>/ (or clicking on the Settings button on the environment page) and adding "CONSTRUCTOR_INSTALLER" to build_artifacts.