Special Filesystems

NHR provides tailored WORK file systems for improved IO throughput of IO intense job workloads.

Default Lustre (WORK)

WORK is the default shared file system for all jobs and can be accessed using the $WORK variable. WORK is accessible for all users and consists of 8 Metadata Targets (MDT’s) with NVMe SSDs and 28 Object Storage Targets (OST’s) on Lise and 100 OST’s on Emmy handling the data. Both using classical hard drives.

Access: $WORK
Size: 8 PiB quoted

Special File System Types

Lustre with striping (WORK)

Some workloads will benefit of striping. Files will be split transparently between a number of OSTs.

Especially large shared file IO patterns will benefit from striping. Up to 28 OSTs on Lise and up to 100 OST’s on Emmy can be used, recommended are up to 8 OSTs for Lise and 32 OSTs on Emmy. We have preconfigured a progressive file layout (PFL), which sets an automatic striping based on the file size.

Access: create a new directory in $WORK and set lfs setstripe -c <stripsize> <dir>
Size: 8 PiB like WORK

Local SSDs

Some Compute Nodes are installed with local SSD storage up to 2 TB on Lise and 480 GB or 1TB (depending on the node) on Emmy.

FastIO

WORK is extended with 4 (Lise)/8 (Emmy) additional OST’s using NVMe SSDs to accelerate heavy (random) IO-demands. To accelerate specific IO-demands further striping for up to 4/8 OSTs is available.

Access:

Lise: create a new directory in $WORK and set lfs setstripe -p flash <dir>

Emmy: Using either $SHARED_SSD_TMPDIR for a job specific folder (like $LOCAL_TMPDIR) oder set the storage pool ddn_ssd for a new directory like for Lise (there the SSD pool is called flash).

Size:

Lise: 55 TiB - quoted

Emmy: 120 TiB - quoted

Finding the right File System

If your jobs have a significant IO part we recommend asking your consultant via support@hlrn.de to recommend the right file system for you.

Local IO

If you have a significant amount of node-local IO which is not needed to be accessed after job end and will be smaller than 2 TB on Lise and 400 GB/1 TB (depending on the node) on Emmy we recommend using $LOCAL_TMPDIR. Depending on your IO pattern this may accelerate IO to up to 100%.

Global IO

Global IO is defined as shared IO which will be able to be accessed from multiple nodes at the same time and will be persistent after job end.

Especially random IO will be accelerated up to 200% using FastIO on Lise.

Performance Comparison of the different File Systems and SSDs for Emmy with IO500

Please remember that we are comparing here a single SSD for the node local SSDs, 43 SSDs for Lustre SSD and 1000 HDDs for Lustre HDD using 32 IO processes per node. For the Lustre filesystems 64 nodes were used to achieve near maximum performance for the Lustre HDD pool. For the Home filesystem with its 120 HDDs only 16 nodes with 10 processes per node were used, as more nodes or processes overloads this small filesystem, resulting in even lower performance.

A typical user job will see lower performance values as there are usually less IO processes. The numbers for the global filesystems indicate the aggregate performance that is distributed across all users.