Cluster Storage Map

Not every filesystem is available everywhere on the cluster, nor is the filesystem performance equal for all nodes. The differences depend on which cluster island a node is a member of and whether it is a login or a compute node. Our general CPU node islands are called “Emmy Phase X”, GPU node islands are called “Grete Phase X”, where X indicates the hardware generation (1, 2, 3, …). Other islands exist for specific institutions/groups or historical reasons (e.g. SCC Legacy).

Which systems can be accessed and the relative performance (limited by the storage system’s performance or the network connection) between each group/island of nodes and each filesystem is shown in the diagram below:

Diagram of the connections between each HPC node group and the storage systems. The storage transfer nodes (gwdu[107-108]) have a slower-medium connection to the GWDG Unix HOME and a very slow connection to the GWDG Archive (AHOME).The MDC login nodes (Emmy Phase 2, glogin[4-8]), RZG login nodes (Grete and Emmy Phase 3, glogin[9-13]), and storage transfer nodes (gwdu[107-108]) have a very slow connection to NHR PERM. All nodes have a slow-medium connection to the Vast filesystems (HOME, NHR Project storage) and CephFS (COLD, SCC Project storage, Workspaces). The MDC login nodes (Emmy Phase 2, glogin[4-8]), SCC Legacy login nodes (gwdu[101-102]), Emmy Phase 2 compute nodes, SCC Legacy compute nodes, and storage transfer nodes (gwdu[107-108]) have very fast connections to Lustre MDC. The RZG login nodes (Grete and Emmy Phase 3,glogin[9-13]), Grete compute nodes, and Emmy Phase 3 compute nodes have a very fast connection to Lustre RZG.

HPC Storage Systems

Connections between each HPC node group/island and the different storage systems, with the arrow style indicating the performance (see key/legend at bottom right).

There are a few important points to note:

  • SCC users have access to parts of the CPU and GPU nodes in the Grete Phase 2 & 3 and Emmy Phase 3 islands (the scc-cpu and scc-gpu partitions) in addition to the SCC Legacy island.
  • Both of our main datacenters have a dedicated Lustre filesystem connected to the cluster islands located there via a very fast network for Workspaces and Temporary Storage.
  • NHR PERM is only accessible from login nodes.
  • The GWDG Tape Archive AHOME is only accessible from the GWDG storage transfer nodes.
  • Temporary Storage for Slurm jobs is always allocated in RAM, on fast local SSDs, and the data center’s Lustre filesystem.
  • The CIDBN, FG, and SOE islands are not shown in the diagram above but have access to the same storage systems with the same relative performance as the SCC Legacy, except that they have their own dedicated BeeGFS filesystems instead of Lustre MDC/RZG.
Info

See CPU Partitions and GPU Partitions for the available partitions in each island for each kind of account.

See Logging In for the best login nodes for each island (other login nodes will often work, but may have access to different storage systems and their hardware will be less of a match).

See Software Stacks for the available and default software stacks for each island.

See Types of User Accounts if you are unsure what kind of account you have.