Cluster Storage Map

Not every data store is available everywhere on the cluster, nor is the filesystem performance equal for all nodes. The differences depend on which cluster island a node is a member of and whether it is a login or a compute node. Our general CPU node islands are called “Emmy Phase X”, GPU node islands are called “Grete Phase X”, where X indicates the hardware generation (1, 2, 3, …). Other islands exist for specific institutions/groups or historical reasons (e.g. SCC Legacy).

Which systems can be accessed and the relative performance of network links between each group/island of nodes and each data store is shown in the diagram below:

Diagram of the connections between each HPC node group and the storage systems. The storage transfer nodes (gwdu[107-108]) have a slower-medium connection to the GWDG Unix HOME and a very slow connection to the GWDG Archive (AHOME).The MDC login nodes (Emmy Phase 2, glogin[4-8]), RZG login nodes (Grete and Emmy Phase 3, glogin[9-13]), and storage transfer nodes (gwdu[107-108]) have a very slow connection to NHR PERM. All nodes have a slow-medium connection to the HOME/Project storage (VAST/CephFS), COLD (CephFS), Workspaces Ceph (CephFS), and Software. The SCC Legacy login nodes (gwdu[101-102]) and SCC Legacy compute nodes very fast connection to the SCRATCH SCC (BeeGFS). The MDC login nodes (Emmy Phase 2, glogin[4-8]), Emmy Phase 2 compute nodes, and storage transfer nodes (gwdu[107-108]) have very fast connections to Workspaces Lustre MDC (Lustre) and SCRATCH SCC (BeeGFS). The RZG login nodes (Grete and Emmy Phase 3,glogin[9-13]), Grete compute nodes, and Emmy Phase 3 compute nodes have a very fast connection to the Workspaces + SCRATCH RZG (formerly known as SCRATCH Grete, Lustre) and and a medium connection to the SCRATCH SCC (BeeGFS).

HPC Storage Systems

Connections between each HPC node group/island and the different storage systems, with the arrow style indicating the performance (see key/legend at bottom right).

There are a few important points to note:

  • SCC users have access to parts of the CPU and GPU nodes in the Grete Phase 2 & 3 and Emmy Phase 3 islands (the scc-cpu and scc-gpu partitions) in addition to the SCC Legacy island.
  • Each island (SCC Legacy, Emmy Phase 2, Grete & Emmy Phase 3) has a separate SCRATCH/WORK data store connected via a very fast network, with slower connections to the other ones.
  • NHR PERM is only accessible from login nodes.
  • The GWDG Tape Archive AHOME is only accessible from the GWDG storage transfer nodes.
  • Temporary Storage for Slurm jobs is always allocated in RAM, on fast local SSDs, and the fastest SCRATCH/WORK available for the node.
  • The CIDBN, FG, and SOE islands are not shown in the diagram above but have access to the same storage systems with the same relative performance as the SCC Legacy, though some also have their own dedicated SCRATCH/WORK.
Info

See CPU Partitions and GPU Partitions for the available partitions in each island for each kind of account.

See Logging In for the best login nodes for each island (other login nodes will often work, but may have access to different storage systems and their hardware will be less of a match).

See Software Stacks for the available and default software stacks for each island.

See Types of User Accounts if you are unsure what kind of account you have.