NHR Modules (nhr-lmod)

The default stack on the whole NHR cluster. This stack uses Lmod as its module system. For the purposes of setting the desired software stack (see Software Stacks), its short name is nhr-lmod. You can learn more about how to use the module system at Module Basics.

To see the available software, run

module avail

The modules for this stack are built for several combinations of CPU architecture and connection fabric to support the various kinds of nodes in the cluster. The right module for the node is automatically selected during module load.

Getting Started with nhr-lmod

On NHR Emmy the new software stack is enabled by default. Just login to glogin-p2.hpc.gwdg.de or glogin-p3.hpc.gwdg.de and use the module avail and module load commands.

On NHR Grete you need to login to glogin-gpu.hpc.gwdg.de and run the following commands since hlrn-tmod is loaded by default:

export PREFERRED_SOFTWARE_STACK=nhr-lmod
source /sw/etc/profile/profile.sh

These commands also have to be put in your slurm batch scripts on NHR Grete to make sure the nhr-lmod stack is used.

Below we have provided some example scripts that load the gromacs module and run a simple test case. You can copy the example script and adjust it to the modules you would like to use.

KISSKI and REACT users can take the Grete example and use --partition kisski or --partition react instead.

#!/bin/bash
#SBATCH --job-name="NHR-Emmy-P2-gromacs"
#SBATCH --output "slurm-%x-%j.out"
#SBATCH --error "slurm-%x-%j.err"
#SBATCH --nodes 1
#SBATCH --ntasks-per-node 96
#SBATCH --partition standard96
#SBATCH --time 60:00

echo "================================ BATCH SCRIPT ================================" >&2
cat ${BASH_SOURCE[0]} >&2
echo "==============================================================================" >&2

module load gromacs/2023.3

export OMP_NUM_THREADS=1

source $(which GMXRC)
mpirun gmx_mpi mdrun -s /sw/chem/gromacs/mpinat-benchmarks/benchPEP.tpr \
	-nsteps 1000 -dlb yes -v 

#!/bin/bash
#SBATCH --job-name="NHR-Emmy-P3-gromacs"
#SBATCH --output "slurm-%x-%j.out"
#SBATCH --error "slurm-%x-%j.err"
#SBATCH --nodes 1
#SBATCH --ntasks-per-node 96
#SBATCH --partition medium96s
#SBATCH --time 60:00

echo "================================ BATCH SCRIPT ================================" >&2
cat ${BASH_SOURCE[0]} >&2
echo "==============================================================================" >&2

module load gromacs/2023.3

export OMP_NUM_THREADS=1

source $(which GMXRC)
mpirun gmx_mpi mdrun -s /sw/chem/gromacs/mpinat-benchmarks/benchPEP.tpr \
	-nsteps 1000 -dlb yes -v 

#!/bin/bash
#SBATCH --job-name="NHR-Grete-gromacs"
#SBATCH --output "slurm-%x-%j.out"
#SBATCH --error "slurm-%x-%j.err"
#SBATCH --nodes 1
#SBATCH --ntasks-per-node 8
#SBATCH --gpus A100:4
#SBATCH --partition grete
#SBATCH --time 60:00

echo "================================ BATCH SCRIPT ================================" >&2
cat ${BASH_SOURCE[0]} >&2
echo "==============================================================================" >&2

export PREFERRED_SOFTWARE_STACK=nhr-lmod
source /sw/etc/profile/profile.sh

module load gromacs/2023.3-cuda

# OpenMP Threads * MPI Ranks = CPU Cores
export OMP_NUM_THREADS=8
export GMX_ENABLE_DIRECT_GPU_COMM=1

source $(which GMXRC)
mpirun gmx_mpi mdrun -s /sw/chem/gromacs/mpinat-benchmarks/benchPEP-h.tpr \
	-nsteps 1000 -v -pme gpu -update gpu -bonded gpu -npme 1

#!/bin/bash
#SBATCH --job-name="NHR-Grete-H100-gromacs"
#SBATCH --output "slurm-%x-%j.out"
#SBATCH --error "slurm-%x-%j.err"
#SBATCH --nodes 1
#SBATCH --ntasks-per-node 8
#SBATCH --gpus H100:4
#SBATCH --partition grete-h100
#SBATCH --time 60:00

echo "================================ BATCH SCRIPT ================================" >&2
cat ${BASH_SOURCE[0]} >&2
echo "==============================================================================" >&2

module load gromacs/2023.3-cuda

# OpenMP Threads * MPI Ranks = CPU Cores
export OMP_NUM_THREADS=12
export GMX_ENABLE_DIRECT_GPU_COMM=1

source $(which GMXRC)
mpirun gmx_mpi mdrun -s /sw/chem/gromacs/mpinat-benchmarks/benchPEP-h.tpr \
	-nsteps 1000 -v -pme gpu -update gpu -bonded gpu -npme 1

Hierarchical Module System

The module system has a Core - Compiler - MPI hierarchy. If you want to compile your own software, please load the appropriate compiler first and then the appropriate MPI module. This will make the modules that were compiled using this combination visible: if you run module avail you can see the additional modules at the top above the Core modules.

The modules built with GCC 11.4.0 and Open MPI are visible by default in the current software revision. We will revisit this decision in 2025 and will probably reduce the number of Core modules significantly and require a module load openmpi before any modules using Open MPI can be loaded.

Supported Compiler - MPI Combinations for Release 24.05

module load gcc/11.4.0
module load openmpi/4.1.6
module avail

module load gcc/13.2.0
module load openmpi/4.1.6
module avail

module load intel-oneapi-compilers/2023.2.1
module load intel-oneapi-mpi/2021.10.0
module avail

module load intel-oneapi-compilers/2023.2.1
module load openmpi/4.1.6
module avail

Adding Your Own Modules

See Using Your Own Module Files.

Spack

Spack is provided as the spack module to help build your own software.

Migrating from HLRN Modules (hlrn-tmod)

For those migrating from HLRN Modules (hlrn-tmod), three important modules were replaced with slightly different modules (in addition to different versions) and several others have renames which are listed in the table below:

Note that some packages are compiled with and without CUDA, and some compiled with CUDA are compiled with multiple versions. In this case, the module name and version will have the form MODULE/VERSION_cuda or MODULE/VERSION_cuda-CUDAMAJORVERSION where CUDAMAJORVERSION is the CUDA major version (e.g. 11 or 12).