AlmaLinux 10 Testing Phase

Our upcoming hardware generation (planned to go online in the second half of 2026) will launch running AlmaLinux 10. Once it has been launched into production, the rest of the HPC compute and login nodes will also be upgraded to the new operating system.

This means we will be upgrading from the RHEL 8 compatible Rocky 8 straight to AlmaLinux 10, skipping Rocky/Alma 9, to reduce the number of potentially breaking changes affecting the users overall. We encourage all users to test their workloads to ensure a smooth transition and to benefit from the new hardware generation as soon as it is put into operation. In addition, we are grateful to receive feedback - both success stories and any obstacles encountered - via the ticket system (please include “AlmaLinux 10” or just “Alma 10” in the subject of your ticket).

Test Nodes

We have prepared test nodes that are already running AlmaLinux 10. These nodes are open to all NHR, SCC, and KISSKI users to test their codes and provide technical feedback. We want to give everyone the chance to make sure their problems are addressed before the new operating system is rolled out cluster-wide (missing essential software, MPI problems, etc.). The test nodes have their own partitions which are:

NHR and KISSKI jobs on these partitions will be “free of charge” (will be accounted as 0 core-hours) during the test phase.

These nodes are shared so multiple users can try them out at once. When the time to upgrade the rest of the system draws closer, more test nodes will be made available.

The Emmy P3 login node glogin13.hpc.gwdg.de is running the new operating system already. It has been removed from the alias glogin-p3.hpc.gwdg.de for the duration of the testing phase.

Please note that the nodes will likely be rebooted quite often, as new software or features are added, and their configurations are fixed and/or adjusted. The installed software might also have to be reinstalled, which requires a re-compilation of your codes if further testing is desired.

Software Changes

Software highlights of the AlmaLinux 10 upgrade:

• System GCC 8 ⇒ 14
• glibc 2.28 ⇒ 2.39
• System Python 3.6 ⇒ 3.12
• Linux kernel (with heavy backports) 4.18 ⇒ 6.12
• OS packages compiled for x86-64-v1 ⇒ x86-64-v3 (AVX2 baseline)
• Podman 5.8.2, which works with caveats

A new software revision is available on the test nodes using the module system. Some of the highlights:

• Updates to many packages
• Main software compiled with
  • GCC 13.4 (GPU nodes)
  • GCC 15.3 (CPU Nodes)
  • Intel OneAPI Compilers 2025.3.2 (Intel CPU Nodes)
  • AMD Optimizing Compilers (AOCC) 5.1.0 (AMD CPU Nodes)
• MPI
  • Intel OneAPI
    • Update 2021.14.0 ⇒ 2021.17.2
  • OpenMPI
    • Update 4.1.7 ⇒ 4.1.8 as well as 5.0.10
    • New OPX provider
      • Required by the upcoming hardware generation
      • Better latency and bandwidth for small transfers
      • Slight degradation of bandwidth for large transfers
• Python 3.13.13
• CUDA 12.9.2 and 13.2.1
• Numerical Libraries
  • Intel MKL 2025.3.1
  • AMD BLIS, libflame, FFTW, Scalapack 5.3
  • OpenBLAS 0.3.33

FUSE and Mount Namespaces

On the login nodes, each user has their own private mount namespace shared between all their sessions. On the compute nodes, each slurm job runs in its own private mount namespace. Each of these private mount namespaces starts with initially-empty /tmp, /var/tmp, and /dev/shm just for the user/job. FUSE is now setup to allow FUSE mounts onto any directory you have write permission to, without the need to first enter a user namespace using unshare. The nodes have several packages for mounting various things via FUSE ready to go:

• ratarmount for mounting many forms of archives (tar, zip, 7z, SquashFS, bind mounting directories, FAT filesystems, ext4, etc.)
• bindfs for bind-mounting directories to other directories
• erofsfuse for erofs
• fuse2fs for ext2/3/4
• fuse-overlayfs for OverlayFS
• squashfuse for SquashFS
• sshfs
• s3fs

There are three important caveats/issues with FUSE you must be careful with:

An example job script setup to receive a SIGUSR1 (signal 10) 300 seconds before running out of time would be:

...
#SBATCH --signal=B:10@300

...

# Do FUSE mounts at /tmp/foo/bar and /tmp/this/that
...

trap SIGUSR1 'pkill -15 -f my-computation.sh ; fusermount -u /tmp/foo/bar ; fusermount -u /tmp/this/that'

./my-computation.sh [ARGS]

fusermount -u /tmp/foo/bar
fusermount -u /tmp/this/that

Notes on OpenMPI

The default OpenMPI 5 module configures the following environment:

FI_PROVIDER=opx
OMPI_MCA_pml=cm
OMPI_MCA_btl=self,vader
OMPI_MCA_mtl=ofi
OMPI_MCA_osc=rdma

The new OPX driver (FI_PROVIDER=opx) will be required for optimal performance on the upcoming hardware and also supports the current generation hardware on Emmy.

The OpenMPI 4 module continues to use the same driver from the previous software revision and can be used as a fallback (the same configuration is impossible with OpenMPI 5 because of the removal of the pt2pt component).

Please let us know about any issues you run into with the new OpenMPI 5 version via Support.

Using PSM2 with OpenMPI 5

If you want to use the older PSM2 driver with OpenMPI 5, you can set the following environment variable:

export FI_PROVIDER=psm2

Because libfabric (ofi) is used, by default two device contexts will be opened for each MPI rank, one for the MPI Matching Transport Layer (mtl), and one for the Byte Transport Layer (btl).

This means that you will run out of device contexts if you use more than 69 MPI ranks per node:

PSM2 can't open hfi unit: 0 (err=23)

Many applications do not need the btl layer and you can simply prevent it from using libfabric:

export OMPI_MCA_btl=self,vader

If you do need this layer (for example for the one-sided-communication functions), you can also enable device context sharing

export PSM2_MULTI_EP=0

Other options with OpenMPI 5

It is possible to use the more generic ob1 component of OpenMPI to get better support for one-sided-communication (MPI_Win_*, MPI_Get, MPI_Put etc.) at the potential cost of performance:

export OMPI_MCA_pml=ob1
export OMPI_MCA_osc=^ucx
export OMPI_MCA_mtl=^ofi
export OMPI_MCA_btl=self,vader,ofi
export OMPI_MCA_btl_ofi_mode=2
export OMPI_MCA_btl_ofi_disable_sep=1

Please let us know immediately if any of these workarounds are necessary for your workload - the manufacturer is interested in these cases and is working with us to improve the support in the opx driver.

Notes on Podman

podman and buildah are installed in the OS images rather than provided in the module stack. They work in principle, but there are significant caveats with their operation. Due to these caveats, Apptainer is usually the better option for building and running containers on the cluster.

Like Apptainer and unlike the defaults for Podman on desktop systems, network namespaces are not created for each container since they are disabled at the OS level for security. Instead the containers run on the host network stack. This means that most of the networking command line options of podman simply do not work and the containers can only bind to ports > 1024 inside.

Like Apptainer, subuids/subgids are not available meaning that the ownership of all contents in the containers are squashed to your user’s UID and primary GID. However, unlike Apptainer, Podman can’t use fakeroot or a similar mechanism to let exectuables “think” they are root when they check for that, even if they could technically work. They will throw errors when they can’t chown files/directories to other UIDs/GIDs. This means that some tools will not work inside the containers run with Podman including the package managers of most Linux distributions.

Podman and Buildah use /tmp/local-share-containers/storage for storing images and containers rather than the usual default $HOME/.local/share/containers/storage, since their storage scheme is incompatible with the NFS mounts we use for home directories. This storage location is a tmpfs mounted at /tmp in the private mount namespace for the job (compute nodes) and user (login nodes). This means image and container storage

• Is not shared between nodes (even in the same job) nor between jobs on the same node
• Is not persistent between jobs
• Is not persistent if you log out of all sessions on a login node and then login again
• Its space counts against the RAM limit of your job on compute nodes and your user on login nodes

Any containers you want to use in a job with Podman/Buildah must be put into the storage of every node in each job, before use. It is generally a bad idea to download container images in each job because any slowdown of the download costs job time and thus core-hours. This is made worse if every node in a multi-node job had to download it separately as they compete for bandwidth, especially if the server hosting the images throttles the downloads. Similarly on login nodes, constant redownloading and rebuilding wastes bandwidth and time. There are two ways around this. The most reliable way is to

1. Save the container images as a TAR file in the oci-archive or docker-archive format on persistent storage like $PROJECT or a workspace via podman --format oci-archive -o PATH IMAGE
2. In a job or on a node that needs it, load it with podman load -i PATH

Another more error-prone way is to load the images in a node (possibly in a job) and then make a backup of /tmp/local-share-containers to an archive (e.g. a TAR file) and then restore it in future jobs and sessions on login nodes as needed.

History of Major Changes

The history of major changes to this page are in the table below so that one can easily see what has changed since the last time one read this page. This does not include changes to the number of nodes in each testing partition.