Spack

Getting started

The official spack documentation can be found at https://spack.readthedocs.io/en/v0.21.1

In order to use Spack, you need to load the corresponding module first. The module name for each software stack that has a Spack module is listed below:

Then, load it by

module load spack-user

module load spack

Please note: Some commands of Spack require extra shell functionality and you need to source the environment script. This will also be shown after you load the spack module:

source $SPACK_ROOT/share/spack/setup-env.sh

Now Spack is ready and you can use spack load command to load necessary software or spack install SPEC to install the software.

Loading software

You can find spack packages that are already installed using spack find:

spack find py-numpy

You need to pick a specific version of those listed to load:

spack load py-numpy target=sapphirerapids

Installing software

In most cases, it is enough to know the package name to install the software. For instance, if you want to install zlib, then you can simply run the following command:

spack install zlib

In general, you need to provide a specification and dependencies for the command spack install, where you can select versions, compiler, dependencies and variants. To learn more about Spack specification, please visit Spack’s documentation.

Hardware Architectures

Since we have multiple CPU architectures, connection fabrics, and GPU architectures on the clusters; it can pay off to optimize your software for the architecture of the nodes it will run on. For example, if you plan on running your software on a Cascadelake node, it can be accelerated by compiling it to use Cascadelake’s AVX512 instructions. A package would be compiled by setting the target in the spec to the desired CPU architecture like:

spack install gromacs target=cascadelake

The spack arch command will print the full CPU and OS architecture/target of the node you are on (e.g. linux-rocky8-sapphirerapids), and spack find will show you what you have built for each architecture (architecture is in the headers). The architecture/target is composed of the operating system (linux), the Linux distribution, and the CPU architecture. Note that the architecture/target does not capture other important hardware features like the fabric (mainly MPI libraries and their dependencies) and CUDA architecture. For CUDA, the cuda_arch parameter should be set to the CUDA compute capability.

Make sure to install the software separately for every architecture you want it to run on from a node with that particular architecture. The easiest way to ensure you are on the right architecture is to start an interactive slurm job on the same partition (and same kind of node if a partition is mixed architecture) you want to use the software on. To learn more about Spack and how to install software you can go through its tutorial at https://spack-tutorial.readthedocs.io/en/latest/

The nodes currently supported for Spack and their architecture and partitions are given in the tables organized by cluster in the sections below. Note that to reduce the total number of separate architectures, some are grouped together and rounded down to the lowest common denominator for CPU architectues and the minimum for CUDA architectue. For example, the lowest common denominator of CPU architectures haswell and broadwell is haswell, and CUDA architectures 70 and 75 is 70.

NHR

SCC

The lowest common denominators by partition are