With the term packaging here I am mostly referring to the act of making a software (compiler, scientific libraries, etc.) available to the users on a shared computing environment (in this post, an HPC cluster).

Distributing software on an HPC cluster, even in case of small instances with less than 20 nodes, is not an easy task. Multiple solutions are available to the admins, some of them are battle tested, some are new and some can be considered on the ‘bleeding edge’ side.

The following is a list that (to the best of my knowledge) summarizes as much as possible the available choices for both tasks.

NOTE: for the sake of simplicity I am only considering GNU/Linux based installations. The examples focus on x86 hardware, but ARM/aarch64 is now mainstream on HPC (e.g. NVIDIA Grace Hopper, AWS Graviton) and most of these methods apply to both. Very specific hardware/software combinations (e.g. Cray) are also not considered here.

Distribution methods

  • Packages: highly dependent on the Linux distribution of choice, it’s a relatively quick (although not always easy) way to install software on the cluster. Most of the problems related to this approach rely on dependency complexity, in particular toward the rest of the OS. After a certain point it will be next to impossible to avoid breaking the installation of the base OS (e.g. Python packages and their dependencies are an easy example).

    A useful workaround, now supporting multiple Linux distributions and processor architectures (not only RHEL/CentOS), is OpenHPC. Note that the original CentOS is end of life; the community RHEL clones are now Rocky Linux and AlmaLinux.

  • NFS export(s): centralized setup, entirely under control of the admins. Easy to setup but not flexible enough: it’s next to impossible for the users to provide software to other users/groups without administrative privileges. On the other hand, admins will spend a lot of time installing multiple versions of compilers/libraries or frameworks, etc.

  • High performance File Systems: FS like Lustre, GPFS, BeeGFS, DAOS and the like provide an additional method to distribute software in a centralized way. It’s not a flexible solution in any case, since the users may end up polluting their directories with multiple neglected installations of different softwares and in some case this may also cause performance issues on some filesystems given the increase of metadata operations involved in dealing with a huge amount of very small files.

  • CVMFS: the CernVM File System (“CernVM-FS” or “CVMFS”) is a tool that allows for efficient global distribution of software and data that does not change frequently. Its biggest user base is within the high energy physics (HEP) community, but it is now also widely used to serve prebuilt binary caches (e.g. EasyBuild or Spack artifacts) as a read-only POSIX mount at /cvmfs.

  • Binary caches and mirrors: instead of building software on the cluster, admins can distribute prebuilt binaries that users download and install. Examples include Spack build caches, Nix binary caches, and CVMFS-backed EasyBuild/Spack software. This sharply reduces build time and is increasingly the default for large shared stacks.

Module systems

All the methods above describe how software gets to the cluster; the module system describes how users access it. Environment Modules and, more commonly on HPC, Lmod let users toggle software versions with module load/module swap without altering the base OS. Most of the build tools below emit module files, so the canonical HPC stack is the triad EasyBuild + Lmod + CVMFS: EasyBuild builds the software, Lmod exposes it, and CVMFS distributes the read-only payload.

Build Tools

Third party tools to build and manage HPC software outside of the ones provided by a Linux distribution. These are build and management frameworks whose output is typically a set of module files (see Module systems):

  • Conda creates isolated, versioned environments; conda-forge is the main community channel and mamba is the standard fast solver.
  • EasyBuild (v5.x) remains the de facto HPC build framework. It pairs with Lmod and is the engine behind EESSI, a standardized environment for toolchains and scientific software.
  • Spack emphasizes a self-contained, relocatable install tree and first-class binary distribution via build caches and mirrors, including OCI/Docker registries and CVMFS.

Package managers

For general software installations, the following tools support versioning, rollbacks and binary reproducibility:

  • Guix is a transactional, per-user package manager with bit-for-bit reproducible builds. The Guix-HPC effort (now including the Guix-Science channel) packages HPC software and supports cluster-wide installs as well as relocatable tarballs via guix pack -RR.
  • Nix has become much more mainstream on HPC. It can run as a full OS (NixOS), as a per-user install on shared nodes, and it leverages binary caches; serving the Nix store read-only over CVMFS is a common pattern (see the Compute Canada reference below).

Containers

Traditional Linux containers are really just ordinary processes on a Linux system. These groups of processes are isolated from other groups of processes using resource constraints (control groups [cgroups]), Linux security constraints (Unix permissions, capabilities, SELinux, AppArmor, seccomp, etc.), and kernel namespaces (PID, network, mount, etc.).

While containers were not developed with HPC as the main case study, they are a strong fit for packaging and distributing software on a cluster. On HPC the runtime must be rootless and unprivileged (no daemon, no privilege escalation) and must share GPU and MPI access with the host. The two tools that best meet these constraints are Apptainer (formerly Singularity) and CharlieCloud.

Apptainer is the de facto HPC container runtime. Renamed from Singularity in 2022 and now a Linux Foundation project, it runs unprivileged, uses the single-file SIF format, is compatible with Docker/OCI images, and has first-class GPU and MPI support. It pulls directly from OCI registries (Docker Hub, GHCR, Harbor), which have become the universal distribution format. Docker itself is generally not HPC-friendly because it requires root or elevated privileges. CharlieCloud takes a lighter approach with no daemon.

Today the dominant container use case is AI/ML workloads (CUDA, ROCm, PyTorch, LLM serving), where containers plus CVMFS are the standard way to ship heavy, fast-moving stacks.

An interesting example of using containers to distribute software is the ATLAS experiment’s approach1.

Resources

  1. Building and using containers at HPC centres for the ATLAS experiment (CHEP 2018, PDF) ↩

  2. 10 years of EasyBuild, K. Hoste (HPCKP 2019, PDF slides) ↩

  3. EESSI open-access paper ↩