Installing SSH Clients

Instructions for installing the most popular SSH clients on various operating systems.

Linux

On Linux, your desktop environment almost always has a terminal, typically with a name that includes a word like “term”, “terminal”, “console”, or some alternate spelling of them (e.g. Konsole on KDE). The OpenSSH client is usually already installed. To check if it is, pull up a terminal and see what the following command returns:

ssh -V

If it prints something like OpenSSH_9.2p1 [...], it is already installed. Otherwise, use your package manager to install it; it is usually called openssh-client, openssh-clients, or openssh depending on your Linux distribution. The respective command to install it from the terminal is given for several popular distributions:

sudo apt install openssh-client

sudo dnf install openssh-clients

sudo yum install openssh-clients

sudo pacman -S openssh

Mac

Mac OS X and newer already have a terminal and OpenSSH client installed, so nothing more has to be done. The builtin terminal program’s name is Terminal. If you need X11 forwarding in your terminal, you will additionally need to install and use XQuartz. If you are looking for a very powerful terminal emulator, check out iTerm2.

Windows

There are 3 popular options, each detailed below. Note that only MobaXterm provides X11 forwarding for running remote applications with a GUI.

OpenSSH (Windows 10 or newer)

The already installed PowerShell (or the classic cmd.exe) provides the terminal. They should be listed in the Start menu. To check if OpenSSH is installed, run

ssh --version

which will print the OpenSSH client’s version if it is present, and fail if it isn’t installed. If it is not installed, re-run PowerShell as an administrator (right click on it’s Start menu entry to see this option) and install it with

Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0

Then confirm that it works with

ssh --version

Additional instructions can be found in Microsoft’s documentation.

Please see the SSH Troubleshooting section if you encounter problems.

MobaXterm

MobaXterm is a popular SSH client and terminal combination supporting tabs and X11 forwarding. Go to it’s website to download and install it.

PuTTY

PuTTY is another popular SSH client and terminal combination. Go to it’s website to download and install it.

Generating SSH Keys

After making sure your SSH client is installed, the next step is to generate your SSH keys. SSH keys are used to authenticate your client to the cluster frontend nodes as opposed to a password, as well as getting the encrypted session going. After generating the keys, you have to upload your public key so you can authenticate to the cluster in the future.

SSH Key Basics

An SSH key has two parts: the public key that you can share, and the private key which you must protect and never share. In all clients other than PuTTY (and those that use it), the two parts have the same filename except that the public key file has the added extention .pub for public. In PuTTY, the private key has the extention .ppk and the public key can be saved with an arbitrary extension. The best ways to protect the private key is to either encrypt it or to store it on a security key/card with a PIN.

Generate SSH Key

If your client supports it, you should generate an ed25519 key since it is both fast and secure. You can also use ed25519-sk if you have and want to use a compatible FIDO2 security key for 2FA (Second Factor Authentication). Otherwise, you should create a 4096-bit rsa key which is a pretty much universally supported and safe but much larger and slower fallback. See the FAQ page on keys for more information on these keys. Instructions for generating keys for several clients are given below.

MobaXterm

1. Open MobaXterm
2. Click “Start local terminal server”
3. Generate an SSH key the same way as for an OpenSSH client following the instructions below.

OpenSSH in Terminal (Linux, Mac, Windows PowerShell)

To generate a key with the filename KEYNAME (traditionally, one would choose ~/.ssh/id_NAME where NAME is a convenient name to help keep track of the key), you should run the following in your terminal on your local machine

ssh-keygen -t ed25519 -f KEYNAME

ssh-keygen -t rsa -b 4096 -f KEYNAME

and provide a passphrase to encrypt the key with. Choose a secure passphrase to encrypt your private key that you can remember but others cannot figure out! Then the file ~/.ssh/id_NAME is your private key and ~/.ssh/id_NAME.pub is your public key. Your terminal should look something like the following:

foo@mylaptop:~> ssh-keygen -t ed25519 -f ~/.ssh/id_test
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/foo/.ssh/id_test
Your public key has been saved in /home/foo/.ssh/id_test.pub
The key fingerprint is:
SHA256:54NGZLI2MQSowPoqviWFDlJ5S5KHtDwcbjPdHHaEtUY foo@mylaptop
The key's randomart image is:
+--[ED25519 256]--+
|. o...++E        |
|.*.B =.+ .       |
|o./ = * =        |
|oo.O . O         |
|oo .. + S .      |
|+ o  . o +       |
| + .    o o      |
|o o    .   .     |
|oo.              |
+----[SHA256]-----+
foo@mylaptop:~>

PuTTY

SSH keys are generated by the program PuTTYgen. To create a key pair, follow these steps:

1. Open PuTTYgen
2. Set the key parameters at the bottom to either “EdDSA” and “Ed25519 (255 bits)” for ed25519 (best) or “RSA” and “4096 bits” for rsa (fallback).
3. Click Generate and follow the instructions to generate the key
4. Enter a passphrase to encrypt the key. Choose a secure passphrase that you can remember but others cannot figure out!
5. Click both Save private key and Save public key to save the respective key to disk. Note that PuTTY uses a different storage convention than other clients.

Uploading SSH Keys

Now that you have generated your SSH keys, you should upload the public key (NOT the private key) to the authentication system.

Copy SSH Key

First, you must get the public key in the form to upload and copy it. The text you will upload should look something like:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEgOP7sQ2YydiyHVjFVCzBcX20lM10U0wPKNtY9sUu8q foo@mylaptop

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDMd+BFyyJ1d8bKznh8FmKhguSLN/6x2B3mK1zxjiw9Rq5KA4dPGNGULLUhwEd1oracXdi/sMbK9ZduCYt7L19r/K2v4OS9T4IGdMYWR1EbIvFkAvgnnZ2Mo0zDpQtDEn/sbJXbgfexc+ymSXAehX6cSn+7Hq8beBX0OFfcIyHPm15VeI0S0sxn5SEjw78lFiNRFlFWphtGflr7OLQUugXBNqailduGWswbpXBp622sdjI3Cj7DHnJh/7GQhAuGXvAedDEcPrRjYXPIY4nQ3KxtevPDwye4+PBO97FwAd02QQXOrenFhXqbAuG0+rEtCWii/ETKB1BYWmXW+atyMdKT18+nW34VqOeNDoQuwXwri1TrqYo4XwjKB33rgAwsYdUVBVIGxc8kyrSsGEiLHKgl18A6OST0ATKXmuhNBt6PjTrU/tV80YAZD2ACGmhyB2plR+p/mVURcv7KuHyclJ/dnk7i/5Iw4PMQBfvALuUho9sIk9EX09P+zvC8TpcTS5gesYJGvocuIF+iVjyQQmUqKquR/5C+mUhtJ2Iqliks3mIkVzzzoEPNWc/Hz50Iu0VeW/YggZhllFHjzeAinE+yl2lpQzD+1bFgruXSiAkiN/QUu3WEI/HDX4g5vRurBKd/wzK201ygOm0wYCthJzQKKM7R5TZYHitXC05omf35Pw== foo@mylaptop

If your SSH key is stored in the same style as OpenSSH (whether made by OpenSSH, MobaXterm, etc.), open your public key file in a plaintext editor such as Kate or Gedit (Linux), TextEdit (Mac), Notepad (Windows), VSCode, Vim, Emacs, etc. From there, copy your public key to the clipboard so you can paste it for uploading.

If you generated your SSH key with PuTTYgen, open the .ppk file in PuTTYgen by clicking Load. The text to copy is highlighted in the screenshot below

Upload Key

How you upload your SSH public key to your account depends on the kind of account you have. To determine your upload method, consult the table below and then go to the respective section:

Academic Cloud Upload

First, go to Academic Cloud, which is shown in the screenshot below:

Click “Login” and then proceed to enter your primary email address with GWDG (for academic institutions serviced direcly by GWDG like the MPG and Georg-August university, this will be your institutional email address) or your institution if it is part of the federated login or the email address you used if you created an account on this page previously, as shown in the screenshot below. Then enter your password and do any 2FA/MFA you have enabled.

After you have logged in, you will get a screen like the one below. Click on the option menu next to your name at the top-right and select “Profile” to get the your account settings.

In the Account settings, you will see information about your account like your username on the cluster, Academic ID number, email addresses you have registered, etc. as shown in the screenshot below. Click “Security” in the left navigation menu (pointed at by the red arrow).

In the security panel, towards the bottom, there is a box for your SSH Public Keys as shown in the screenshot below. The SSH public keys you have uploaded so far are listed here. Click on the “ADD SSH PUBLIC KEY” button pointed at by the red arrow to add your key. Then, in the window that pops up (center of the screenshot), paste the key you copied earlier and optionally add a suitable comment. Click “ADD” to upload it.

HLRN Upload

If you have an old HLRN username (like for example nibjdoe or beijohnd), you can login to AcademicID with the email <HLRN username>@hlrn.de. Your account does not have a password set by default, please use the password reset function of AcademicCloud. Please also use <HLRN username>@hlrn.de as the address to send the new password to. It will be forwarded to your real email address you used to register the original HLRN account, but directly specifying your real email won’t work!

Afterwards you can upload your SSH key just like other AcademicCloud users. Please note that your account has a different username in AcademicCloud, which you will see under AcademicID, but it is not relevant or used anywhere else.

If you want to use the NHR@ZIB systems in Berlin that are not run by GWDG, please visit https://portal.nhr.zib.de/

Deleting lost/stolen SSH keys

To delete an old key that you don’t need any more, have lost or that might have been compromised, start by following the same steps as detailed above. In the last step, at the point where you can add new keys, you will see a list of your existing ones with a trash can icon next to each (for example the login key in the last screenshot). Click it and confirm the deletion in the following dialog.

Configuring SSH

OpenSSH, sftp, rsync, VSCode, …

The OpenSSH configuration file is a plain text file that defines Hosts with short, easy to type names and corresponding full DNS names (or IPs) and advanced configuration options, which private key to use, etc. This file being picked up and read by multiple SSH clients or other applications that use the SSH protocol to connect to the cluster. Most clients let you override the defaults in this file for any given login.

Config file location

It is usually located in your home directory or user profile directory under .ssh/config, i.e.

~/.ssh/config
/home/$USERNAME/.ssh/config

~/.ssh/config
/User/$USERNAME/.ssh/config

%USERPROFILE%\.ssh\config
C:\Users\your_username\.ssh\config

OpenSSH config file format

After general options that apply to all SSH connections, you can define a line Host my_shorthand, followed by options that will apply to this host only, until another Host xyz line is encountered. See the OpenSSH documentation for a full list of available options.

Simple configuration examples

Here are different blocks of configuration options you can copy-paste into your config file. You can mix-and-match these, choose the ones you need. Just make sure to replace User xyz with your respective username and if necessary, the path/filename of your private key.

Emmy / Grete (NHR and SCC)

While “Emmy” was originally the name of our NHR/HLRN CPU cluster, we will sometimes refer to it as a generic term for all our CPU partitions. For example, the scc-cpu partition is integrated with Emmy Phase 3 from a technical standpoint, thus Emmy-p3 is the recommended login for SCC users. The same applies to “Grete”, which can be used as the generic login if you want to use any of our GPU partitions.

Host Emmy
	Hostname glogin.hpc.gwdg.de
	User u12345
	IdentityFile ~/.ssh/id_ed25519

Host Emmy-p2
	Hostname glogin-p2.hpc.gwdg.de
	User u12345
	IdentityFile ~/.ssh/id_ed25519

Host Emmy-p3
	Hostname glogin-p3.hpc.gwdg.de
	User u12345
	IdentityFile ~/.ssh/id_ed25519

Host Grete
	Hostname glogin-gpu.hpc.gwdg.de
	User u12345
	IdentityFile ~/.ssh/id_ed25519

KISSKI

Host KISSKI
	Hostname glogin-gpu.hpc.gwdg.de
	User u56789
	IdentityFile ~/.ssh/id_ed25519

Legacy SCC

The legacy SCC login nodes are not reachable from outside GÖNET. You can use the generic login nodes as jumphosts, to avoid the need for VPN if you want to connect from the outside.

Host jumphost
	Hostname glogin.hpc.gwdg.de
	User jdoe1
	IdentityFile ~/.ssh/id_ed25519

Host SCC-legacy
	Hostname login-mdc.hpc.gwdg.de
	User jdoe1
	IdentityFile ~/.ssh/id_ed25519
	ProxyJump jumphost

Advanced configuration examples

Here are more advanced example configuration for all our clusters with convenient shortcuts for advanced users. Adapt to your needs. To understand this, remember ssh parses it’s config top-down and first-come-first-serve.

That means, if for a given Host the same option appears multiple times, the first one counts and subsequent values are ignored. But for options that were not defined earlier, values will apply from successive appearances. Note that if Hostname is not specified, the Host value (%h) is used.

All frontend nodes, individually specified

# NHR, SCC Project-Portal and KISSKI users
Host glogin*
	Hostname %h.hpc.gwdg.de
	User u12345
	IdentityFile ~/.ssh/id_ed25519


# Use the main login nodes as jumphosts to restricted login nodes
Host jumphost
	Hostname glogin.hpc.gwdg.de
	User jdoe1
	IdentityFile ~/.ssh/id_ed25519

# legacy SCC login nodes
Host gwdu101 gwdu102 login-mdc
	Hostname %h.hpc.gwdg.de
	User jdoe1
	IdentityFile ~/.ssh/id_ed25519
	ProxyJump jumphost

# CIDBN
Host login-dbn*
	Hostname %h.hpc.gwdg.de
	User u23456
	IdentityFile ~/.ssh/id_ed25519
	ProxyJump jumphost

# Cramer/Soeding
Host ngs01
	Hostname ngs01.hpc.gwdg.de
	User u34567
	IdentityFile ~/.ssh/id_ed25519
	ProxyJump jumphost

Complex example using config file tricks

Host myNHRproject1
	User u10123
Host myNHRproject2
	User u10456
Host NHR myNHRproject1 myNHRproject2
	Hostname glogin.hpc.gwdg.de
	IdentityFile ~/.ssh/id_ed25519
Host kisski
	Hostname glogin-gpu.hpc.gwdg.de
	User u10789
	IdentityFile ~/.ssh/id_ed25519
Host glogin* glogin-gpu glogin-p3
	Hostname %h.hpc.gwdg.de
	User nimjdoe
	IdentityFile ~/.ssh/id_ed25519

Host jumphost
	Hostname glogin.hpc.gwdg.de
Host SCC-legacy
	Hostname login-mdc.hpc.gwdg.de
Host SCC-legacy gwdu101 gwdu102 jumphost
	Hostname %h.gwdg.de
	User jdoe1
	IdentityFile ~/.ssh/id_ed25519
Host SCC-legacy gwdu101 gwdu102
	ProxyJump jumphost

Logging In

Here, you’ll find DNS names for our login nodes, sorted by cluster, their hostkey fingerprints and some examples showing how you can connect from the command line.

The Login Nodes

Names and Aliases

The proper DNS names of all login nodes grouped by cluster island and their aliases are provided in the table below. General CPU node islands are called “Emmy Phase X” where X indicates the hardware generation (1, 2, 3, …). General GPU node islands are called “Grete Phase X” where X indicates the hardware generation (1, 2, 3, …). Other islands exist for specific institutions/groups or historical reasons (e.g. SCC Legacy). It is best to use the login nodes for the island you are working with. Other login nodes will often work (assuming you have access), but may have access to different storage systems and their hardware will be less of a match. For square brackets with a number range, substitute any number in the range.

SSH key fingerprints

When first connecting, SSH will warn you that the host’s key is unknown and to make sure that it is correct, in order to avoid man-in-the-middle attacks. The following table contains all SHA-256 fingerprints of the host keys of the login nodes and jumphosts, arranged by key algorithm:

Example Logins with OpenSSH

Logging into an Emmy Phase 3 login node

With the .ssh/config file setup as in the Simple configuration examples, you can run just ssh Emmy-p3. These are the recommended login nodes for SCC users and NHR users who use the Sapphire Rapids nodes (e.g. the standard96s partition). As an SCC user, your terminal session could look like

jdoe1@laptop:~> ssh Emmy-p3
Enter passphrase for key '/home/jdoe1/.ssh/id_ed25519':
Loading software stack: gwdg-lmod
Found project directory, setting $PROJECT_DIR to '/projects/scc/GWDG/GWDG_AGC/scc_agc_test_accounts/dir.project'
 __          ________ _      _____ ____  __  __ ______   _______ ____
 \ \        / /  ____| |    / ____/ __ \|  \/  |  ____| |__   __/ __ \
  \ \  /\  / /| |__  | |   | |   | |  | | \  / | |__       | | | |  | |
   \ \/  \/ / |  __| | |   | |   | |  | | |\/| |  __|      | | | |  | |
    \  /\  /  | |____| |___| |___| |__| | |  | | |____     | | | |__| |
  ___\/__\/   |______|______\_____\____/|_|__|_|______|    |_|  \____/
 |__   __| |  | |  ____|  / ____|/ ____/ ____|
    | |  | |__| | |__    | (___ | |   | |
    | |  |  __  |  __|    \___ \| |   | |
    | |  | |  | | |____   ____) | |___| |____
    |_|  |_|  |_|______| |_____/ \_____\_____|

 Documentation:  https://docs.hpc.gwdg.de
 Support:        hpc-support@gwdg.de

PARTITION    NODES (BUSY/IDLE)     LOGIN NODES
medium             95 /    0     login-mdc.hpc.gwdg.de
scc-cpu            49 /    0     glogin-p3.hpc.gwdg.de
Slurm load last updated 11.8 seconds ago
Your current login node is part of glogin-p3
u12345@glogin11 ~ $

If you are not using a configured Host entry (not recommended) or want to know how to connect “manually” (useful for troubleshooting), you can run:

ssh u12345@glogin-p3.hpc.gwdg.de -i ~/.ssh/id_ed25519

Make sure to replace u12345 with your actual username and ~/.ssh/id_ed25519 with the path to your private key (if you have not used the default).

Logging into a Grete (GPU) login node

With the .ssh/config file setup as in the Simple configuration examples, you can run just ssh Grete. As an NHR user, your terminal session could look like

jdoe1@laptop:~> ssh Grete
Enter passphrase for key '/home/jdoe1/.ssh/id_ed25519':
Loading software stack: gwdg-lmod
Found scratch directory, setting $WORK to '/scratch/usr/nimjdoe'
Found temporary files directory, setting $TMPDIR to '/scratch/tmp/nimjdoe'
 __          ________ _      _____ ____  __  __ ______   _______ ____
 \ \        / /  ____| |    / ____/ __ \|  \/  |  ____| |__   __/ __ \
  \ \  /\  / /| |__  | |   | |   | |  | | \  / | |__       | | | |  | |
   \ \/  \/ / |  __| | |   | |   | |  | | |\/| |  __|      | | | |  | |
    \  /\  /  | |____| |___| |___| |__| | |  | | |____     | | | |__| |
  _  \/ _\/  _|______|______\_____\____/|_|  |_|______|____|_|__\____/
 | \ | | |  | |  __ \     ____    / ____\ \        / /  __ \ / ____|
 |  \| | |__| | |__) |   / __ \  | |  __ \ \  /\  / /| |  | | |  __
 | . ` |  __  |  _  /   / / _` | | | |_ | \ \/  \/ / | |  | | | |_ |
 | |\  | |  | | | \ \  | | (_| | | |__| |  \  /\  /  | |__| | |__| |
 |_| \_|_|  |_|_|  \_\  \ \__,_|  \_____|   \/  \/   |_____/ \_____|
                         \____/

 Documentation  https://docs.hpc.gwdg.de   Support nhr-support@gwdg.de
PARTITION    NODES (BUSY/IDLE)     LOGIN NODES
grete:shared       53 /    9     glogin-gpu.hpc.gwdg.de
grete-h100          4 /    0     glogin-gpu.hpc.gwdg.de
Your current login node is part of glogin-gpu
[nimjdoe@glogin9 ~]$

If you are not using a configured Host entry (not recommended) or want to know how to connect “manually” (useful for troubleshooting), you can run:

ssh u12345@glogin-gpu.hpc.gwdg.de -i ~/.ssh/id_ed25519

Make sure to replace u12345 with your actual username and ~/.ssh/id_ed25519 with the path to your private key (if you have not used the default).

Logging into a legacy SCC login node

With the .ssh/config file setup as in the Simple configuration examples, you can just run ssh SCC. Your terminal session could look something like

jdoe1@laptop:~> ssh SCC
Enter passphrase for key '/home/jdoe1/.ssh/id_ed25519':
Last login: Tue May 20 16:27:44 2025 from 10.45.112.11
Loading software stack: gwdg-lmod
 __          ________ _      _____ ____  __  __ ______   _______ ____
 \ \        / /  ____| |    / ____/ __ \|  \/  |  ____| |__   __/ __ \
  \ \  /\  / /| |__  | |   | |   | |  | | \  / | |__       | | | |  | |
   \ \/  \/ / |  __| | |   | |   | |  | | |\/| |  __|      | | | |  | |
    \  /\  /  | |____| |___| |___| |__| | |  | | |____     | | | |__| |
  ___\/__\/   |______|______\_____\____/|_|__|_|______|    |_|  \____/
 |__   __| |  | |  ____|  / ____|/ ____/ ____|
    | |  | |__| | |__    | (___ | |   | |
    | |  |  __  |  __|    \___ \| |   | |
    | |  | |  | | |____   ____) | |___| |____
    |_|  |_|  |_|______| |_____/ \_____\_____|

 Documentation:  https://docs.hpc.gwdg.de
 Support:        hpc-support@gwdg.de

PARTITION    NODES (BUSY/IDLE)     LOGIN NODES
medium             94 /    0     login-mdc.hpc.gwdg.de
scc-cpu           236 /   79     glogin-p3.hpc.gwdg.de
scc-gpu            21 /    5     glogin-gpu.hpc.gwdg.de
Your current login node is part of login-mdc
[scc_project] u12345@gwdu101 ~ $

When using a jumphost, you may have to enter the passphrase for your private key twice (once for the jumphost and once for the actual login node).

If you are not using a configured Host entry (not recommended) or want to know how to connect “manually” (useful for troubleshooting), here is how you would do so (using a jumphost):

ssh jdoe1@login-mdc.hpc.gwdg.de -i .ssh/id_ed25519 -J jdoe1@glogin.hpc.gwdg.de

Or, if you are using an older SSH client:

ssh jdoe1@login-mdc.hpc.gwdg.de -i .ssh/id_ed25519 -o ProxyCommand="ssh -i .ssh/id-rsa -W %h:%p jdoe1@glogin.hpc.gwdg.de"

Make sure to replace jdoe1 with your actual username.

Logging into a specific node

This is important if you need to

• Reconnect to a tmux or screen session
• Reconnect to a session started by an IDE over SSH
• Use a dedicated login node for your research group
• Use a login node with particular hardware

With the .ssh/config file setup as in the Advanced configuration examples for all nodes configured individually, you would run just ssh NODE where NODE is the name of the node or a suitable alias.

SSH Troubleshooting

Troubleshooting

Various SSH problems and how to solve them are listed below. For general SSH questions not related to problems, see the SSH FAQ page instead.

• "Corrupted MAC" errors on Windows
• Error: "permission denied"
• Warning: unprotected key file

External helpful articles and solutions

• WSL 2 and VPN connection

SSH FAQ

FAQ

Various questions and their answers are listed below. For SSH connection problems, see the SSH Troubleshooting page instead.

• Is password authentication supported?
• What 2FA methods are available?
• What type of user account do I have?
• Why are only ed25519, ed25519-sk, and 4096-bit rsa keys recommended?

Institutional Access to AI Services

Description

In this section, you will find explanations on how to access the AI services in three main steps:

1. selecting the subject matter of the contract
2. selecting the deployment
3. limiting access

Note that you can always discuss with us customization and alternative options.

Contract Selection

• Access to the Open-Weight-Models hosted by GWDG in Göttingen.
• Orthogonal to this offer are external models that you can use but you need to pay for. Based on your choice, we have a different list of contracts and legal requirements that must be fulfilled.

Contracts for Open Weight models hosted by GWDG

Under Contract Documents Internal Models, you will find the necessary documents for using the Open Weight Models hosted by us in Göttingen. Please follow the steps in the instructions and forward the documents to the relevant departments at your institution for review and signature.

In order to use the AI-Basis services, the following documents must be completed, signed and returned:

• performance agreement
• a corresponding data processing contract (AVV)
• Nomination of the Data Protection Officer
• Form for organizational data

You can download the remaining attachments to your contract documents.

Contracts for optional access to external models

If you want to opt in for access to the external models you find the corresponding documents under Contract Documents Internal and External Models. The documents differ in order to additionally allow the use of external models. Access to the internal models is included.

Entry point from DFN

If you wish to access our AI services via our partnership with the DFN, you must also sign the corresponding contracts between you, the GWDG and the DFN. Please contact verkauf@gwdg.de.

Entry point from KI:Connect

The default approach for KI:Connect is to use open weight models via API endpoint. Legally there is NO need to make an AVV with us but it useful in case someone accidentally puts in GDPR personal data. Institutions that use access to external models via API as part of our partnership with KI:Connect require an additional API key. Please contact kisski-support@gwdg.de for this.

Deployment

Generally, it is your choice how to deploy our services:

• Deployment 1) Using the existing Web interface for end users (such as https://chat-ai.academiccloud.de/)
• Deployment 2) Integrating AI APIs into services you deploy locally (such as your own chat interface)

Deployment 1: Using existing web frontends

To use the AI services, it is generally possible to use our web interface. An Academic ID is required.

Deployment 2: Integrating AI APIs into YOUR services

With an Academic ID, you can also request an API key via the KISSKI website (click on “Book” on the page). You can use the API key in another webfrontend, plugin or your own code.

Limiting Access

Restricting Models

We provide a large assortment of Large Language Models (see Available Models. Available models are regularly upgraded as newer, more capable ones are released. Not all of these may be approved for use by your institution. In order to restrict the use to certain models, the authorized person of your institution must inform us in writing of the selection of models.

Restricting Model Access for specific groups

You may specify different user-groups within your institution by assiging functional affiliations (e.g. employee@university.de, student@university, researcher@university) and instructing us which models are available for each user group. In order to gain access to the IdM objects (accounts, groups, shared mailboxes and resources) of your institute or institution in the GWDG identity management system, it is necessary for us to set up special authorizations for the IdM portal for you. Access to the special authorizations forms can be found here.

Limiting external model access by quotas

With your order confirmation you can impose a finanical ceiling for the usage of the external models. As a result, total consumption can be reduced to the corresponding monetary value of this limit.

Apply for a Compute Project

Application Types

You can apply for two different types of projects, to a Test Project and to a Compute Project with the following properties.

NHR Starter vs Test projects

Please refer to the notice above on how to apply for an NHR Starter project. You will get access under the same conditions as with a Test Project. However, the additional metadata, such as your field of research, short project description and required software will help us to better understand your requirements.

Application Process

Project proposals for a Compute Project have to be submitted via the portal JARDS.

• Our Guide to filling out offers assistance to go through JARDS.
• One significant point on JARDS is to upload the main proposal document. To prepare this main proposal document we suggest to use the Proposal Format which is used for NHR@ZIB and NHR-NORD@Göttingen since both centers collaborate for the technical and scientific evaluation of compute time projects.
• If you plan to use the NHR systems at both NHR centers (NHR@ZIB and NHR-Nord@Göttingen), it is recommended to submit two analogous proposals via JARDS. The technical review will then be done by each center and the scientific review is done jointly. Usually, the two proposals will then only have to differ in the amount of required resources and how they are mapped to the work packages. If you submit a pair of proposals like this, please make sure to state this under Remarks, including the mutual project title, unless the projects are named identically anyway.

Deadlines and schedule

Compute project proposals can be submitted via JARDS at any time. The review process takes place in the quarter corresponding to the submission deadline: January 15 (23:59:59 CET), April 15 (23:59:59 CEST), July 15 (23:59:59 CEST), and October 15 (23:59:59 CEST). These dates apply to NHR-Nord@Göttingen and extend the default NHR deadlines (first day of each quarter).

Approved projects begin at the first day of the quarter following the review phase and are granted a duration of 12 months.

Proposal Format

Please submit the complete project proposal for a compute project at the two NHR centers, NHR@ZIB and NHR@Göttingen, via JARDS. A project proposal contains four parts which are described in the sections below.

1. Meta Data
2. Main Proposal
3. Public Abstract
4. Signed Proposal Summary

After submission, the proposal is reviewed by the external Scientific Board. Each reviewer obtains access to

• the project proposal,
• the history of previous projects,
• contact data of the consultant, and
• aggregated statistics for usage of computung time on the NHR systems.

Meta Data

During the course of online submission, the project proposer needs to provide some metadata. This includes information about

• the principal investigator (scientist holding a PhD),
• the person in charge of submitting the proposal,
• project partners,
• other evaluations and funding for the project,
• usage of other compute resources,
• project lifetime,
• software requirements (please check availability with your consultant before),
• requirements in units core hour (at least 1200 k core hour per year, see Accounting), and
• storage requirements.

Main Proposal

The main proposal

• needs to be uploaded as a prepared document in PDF format,
• is written in English,
• should be written in LaTeX based on the proposal_template_v1.5.0g.zip. This template includes all relevant aspects like project description, computational facets and resource estimation. The LaTeX template includes both proposal types: initial and follow-up.

We recommend to use the following layout:

• Initial proposal: main_proposal_initial.pdf
• Follow-up proposal: main_proposal_follow_up.pdf

The LaTeX template / PDF samples internally differentiate between a normal and a short version in case of whitelist status. For more details, look inside the template/samples.

Whitelisting

Whitelist status is granted by the scientific review board under two conditions:

• First, you apply for a “Normal project” with less than 5M core-hours per quarter (see Apply for a Compute Project).
• Secondly, you are part of an active DFG/BMBF/NHR/GCS or EU project, which explicitly describes HPC resource requirements. As evidence, you need to upload the corresponding project proposal and its review report to the online portal.

Please avoid:

• Imprecise or incomprehensible estimation of requested computational resources (especially core-hours); for example, missing arguments to justify a certain number of N runs instead of a smaller number. One page is our recommended minimum.
• No proof, that the software to be used is suitable and efficient for parallel execution (and parallel I/O) on our current HPC systems architecture. Recycling a scalability demo by a third party is meaningless, without showing that your planned production run is fully comparable to it (algorithm selection within the software, I/O pattern, machine architecture, problem size per core).
• The overall aim and/or motivation behind the project is unclear.
• The applicant lacks HPC/Unix skills and an experienced co-applicant is missing.
• Insufficiency of the applicant’s local resources is not indicated.
• The NHR was not mentioned in relevant publications.
• Cut & paste previous/parallel proposals; instead, refer to these.

In case of questions, please contact your consultant.

Public Abstract

All present, compute projects that have been successfully reviewed by the Scientific Board are listed on the project list. Each proposal for a compute project needs to submit a public abstract in PDF format based on the public abstract template (English/German). It should be generally understandable.

The abstract is written in English or in German and should contain about 2 pages. If you have no project ID yet (in case of an initial proposal) simply keep the default: “abn12345”.

Signed Proposal Summary

By the end of the online submission process, a summary is generated by JARDS which has to be signed and then either sent to the Office of the Scientific Board or reuploaded to JARDS. This also indicates that your application is successfully submitted.

JARDS - Guide to filling out

Please start your application by choosing

NHR Normal, if you want to apply for less than 20 Mio core-hours/year. If you plan to apply for more than 20 Mio core-hours, choose NHR Large (if you apply only for GPU-hours, choose NHR-Large for more than 25 000 GPU-hours/year).

Please note: for “NHR Large”-projects, the “Whitelist” option is not applicable any more.

• If you need a test account check out how to Apply for a Test Account.
• At NHR@Göttingen every personal user account automatically can use computing time (at least 75 kcoreh/quarter).

Go on by choosing NHR@Göttingen

On this page (E-Mail Callback), please enter your e-mail. You will receive a mail with a link to start your application . This e-mail will be used for identification so please use your official e-mail and always the same.

Start the actual application by pressing “New NHR Project Application”
even if you apply for an extension !
(extension will be used in the future if you extend an existing jards-application)
That is, in the current phase there are only “New NHR Projekt Applications”.
If your project is already running at NHR@Göttingen, please provide your project id in the “Remarks” section at the end and also refer to it in the pdf.
YOU MIGHT USE THE SIMPLIFIED TEMPLATE FOR A FOLLOW-UP PROPOSAL – IN THAT CASE PLEASE ADDITIONALLY UPLOAD THE ORIGINAL APPLICATION AS SUPPORTING MATERIAL.

If you are Principal Investigator and Person of contact (PC and PI), you can check “Apply as both, PI and PC”, otherwise please provide the data of both persons.

Complete the data.

The first field on this page depends on whether you’re submitting an Initial Proposal (for a new project) or a Follow-Up Proposal (to apply for more resources and an extended runtime for an already existing project).

“compute period” is (usually): 01.10 – 30.09 (1 year).
You might apply for a shorter period (in full quarters).

Select your DFG classification.
And answer if your project is a collaboration. Please provide all other relevant projects of your GROUP
(that is, projects with the same PI and in the same area).

Please enter the requested computing time in Million Core hours (CPU). Total first, then divided in quarters. Some questions about software etc. follow.

Please answer questions about your usage of Artificial Intelligence (AI) – if you do not use AI, you do not need to answer these questions. If you want to apply for GPU computing time, please enter the values in GPU-hours, that is hours of a single gpu (not a full node).

Also for GPU usage, some questions about the environment have to be answered.

Please enter your planned storage requests on the file systems Home, Work and Perm (this does NOT mean that this storage will be reserved for you). Bigger storage requests should be described even more precise.

These questions should be answered if you have “unusual” I/O requests.

You can upload only 3 PDF Files.

1. the “actual application”, created with the given template.
2. a public abstract (for our web server) - easy to understand.
3. All other documents, e.g. other reviews, publications etc., have to be combined in a single PDF. You might use free tools like PDF-Shuffler to do so. Then, please upload the combined PDF as “Supporting material” .

Please provide your NHR@Göttingen project id here if you have one, e.g. “bem00027”. You might submit any other remarks here.

Finally, you have to sign your application. After pressing “Finalize”, a summary to sign will be created for you.
You might upload it on jards or send it by e-mail to nhr-support@gwdg.de. Of course, you might also send it to:
Wissenschaftlicher Ausschuss der NHR@Göttingen - Geschäftsstelle -
c/o GWDG, Burckhardtweg 4, 37077, Göttingen

AI

Description

Artificial intelligence became an importance tool for many scientific domains. Various problems are tackled using methods from both classical machine learning and modern deep learning.

Access

You can directly request access to our compute resources, either from NHR or the KISSKI project, which is dedicated to this work. All required information can be found under getting an account.

You can also use the chat AI service directly via KISSKI

Applications

Current available modules

Support

Request an API key for the SAIA service from KISSKI.

More

Links to PR website

Chemistry

Description

Computation in chemistry can be done with various software packages, many of which are installed in our software stack.

Access

You can get access to our NHR resourced by following the NHR application process.

Applications

All the applications that can be used for Chemistry listed

Current available modules

• CP2K — A package for atomistic simulations of solid state, liquid, molecular, and biological systems offering a wide range of computational methods with the mixed Gaussian and plane waves approaches. exciting — a full-potential all-electron code, employing linearized augmented planewaves (LAPW) plus local orbitals (lo) as basis set.
• Gaussian — a computational chemistry application provided by Gaussian Inc https://gaussian.com/.
• GPAW — a density functional theory Python code based on the projector-augmented wave method.
• GROMACS — a versatile package to perform molecular dynamics for systems with hundreds to millions of particles.
• NAMD — a parallel, object-oriented molecular dynamics code designed for high-performance simulations of large biomolecular systems using force fields.
• Octopus — a software package for density-functional theory (DFT), and time-dependent density functional theory (TDDFT)
• Quantum ESPRESSO — an integrated suite of codes for electronic structure calculations and materials modeling at the nanoscale, based on DFT, plane waves, and pseudopotentials.
• RELION — REgularised LIkelihood OptimisatioN is a stand-alone computer program that employs an empirical Bayesian approach to refinement of (multiple) 3D reconstructions or 2D class averages in electron cryo-microscopy(cryo-EM)
• TURBOMOLE — TURBOMOLE is a computational chemistry program that implements various quantum chemistry methods (ab initio methods). It was initially developed at the University of Karlsruhe.
• VASP — a first-principles code for electronic structure calculations and molecular dynamics simulations in materials science and engineering.
• Wannier90 — A program that calculates maximally-localised Wannier functions.
• ASE – The Atomic Simulation Environment – a set of tools and Python modules for setting up, manipulating, running, visualizing, and analyzing atomistic simulations.
• LAMMPS – A parallel, classical potential molecular dynamics code for solid-state materials, soft matter and coarse-grained or mesoscopic systems.
• TURBOMOLE – A program package for ab initio electronic structure calculations. (see “module avail turbomole” on the SMP cluster)
• NWChem – A general computational chemistry code with capabilities from classical molecular dynamics to highly correlated electronic structure methods, designed to run on massively parallel machines.
• MOLPRO – A comprehensive system of ab initio programs for advanced molecular electronic structure calculations.
• PLUMED – A tool for trajectory analysis and plugin for molecular dynamics codes for free energy calculations in molecular systems
• CPMD – A plane wave / pseudopotential implementation of DFT, designed for massively parallel ab-initio molecular dynamics.
• BandUP – Band unfolding for plane wave based electronic structure calculations.
• libcurl — curl - a tool for transferring data from or to a server
• libz — A Massively Spiffy Yet Delicately Unobtrusive Compression Library
• nocache — nocache - minimize caching effects in lustre filesystems
• texlive – LaTeX distribution, typesetting system
• git – A fast, scalable, distributed revision control system

Support

Contact us using nhr-support@gwdg.de.

More

Data Science

Description

Data science is a broad field and we offer a wide variety of software to accomodate as many of the users needs as possible

Access

You can get access to our NHR resourced by following the NHR application process.

Applications

Current available modules

• AEC library — Adaptive Entropy Coding library
• CDO — The Climate Data Operators
• ECCODES — ECMWF application programming interface
• HDF5 Libraries and Binaries — HDF5 - hierarchical data format
• libtiff — A software package containing a library for reading and writing _Tag Image File Format_(TIFF), and a small collection of tools for simple manipulations of TIFF images
• NCO — The NetCdf Operators
• netCDF — Network Common Data Form
• Octave — Add short Excerpt. This will be included in the Software list
• pigz — A parallel implementation of gzip for modern multi-processor, multi-core machine
• PROJ — Cartographic Projections Library
• R — R - statistical computing and graphics
• Szip — Szip, fast and lossless compression of scientific data
• UDUNITS2 — Unidata UDUNITS2 Package, Conversion and manipulation of units
• Boost – Boost C++ libraries
• CGAL – The Computational Geometry Algorithms Library
• nocache — nocache - minimize caching effects in lustre filesystems
• texlive – LaTeX distribution, typesetting system
• git – A fast, scalable, distributed revision control system

Support

Contact us using nhr-support@gwdg.de.

More

Digital Humanities

Description

Digital Humanities is a broad field and we offer a wide variety of software to accommodate as many of the users needs as possible

Access

You can request access to these services via SCC, NHR or KISSKI and this is explained here.

Applications

• Jupyter Notebooks using GPUs
• Apptainer
• Python
• FFMPEG

Current available modules

Support

Contact us using our support.

More

Earth Science

Description

Earth science is a broad field, and we offer a wide variety of software to accommodate as many of the users needs as possible

Access

You can get access to our NHR resourced by following the NHR application process.

Applications

Current available modules

Support

Contact us using nhr-support@gwdg.de.

More

Engineering

Description

Many engineering challenges involve solving multiple equations which varying numbers of constants and physical properties. To make this manageable, many software packaged exists and we offer support for many of them.

Access

You can get access to our NHR resourced by following the NHR application process.

Applications

Current available modules

• Ansys Suite — The full Ansys Academic Multiphysics Campus Solution is available, e.g. Mechanical, CFX, Fluent, LS-Dyna, Electronic, SCADE (but not Lumerical, GRANTA).
• Foam-extend — The Open Source CFD Toolbox
• OpenFOAM — An object-oriented Computational Fluid Dynamics(CFD) toolkit
• STAR-CCM+ — A Package for Computational Fluid Dynamics Simulations
• ParaView — An interactive data analysis and visualisation tool with 3D rendering capability
• git – A fast, scalable, distributed revision control system

Support

Contact us using nhr-support@gwdg.de.

More

Forest

Forests play a crucial role in our ecosystem and are, therefore, being researched on a large scale. Modern approaches are based on big data (e.g., analyzing satellite imagery or single-tree-based simulations). These approaches require significant computing resources, which we can provide with our HPC systems.

Our general motivation and topics of interest can be found on our community page. This page describes tools, data, and computing resources that we believe can be useful based on our experience. If you work in forest science and have ideas or requests, please do not hesitate to contact us via our ticket system or the responsible person listed on the community page.

Forest workloads often fit the workflow of a typical HPC job. Therefore, it is recommended that you start with our general documentation on how to use the cluster.

Following are some of our services and tools highlighted, which we think are useful to consider when doing forest research on our HPC system.

JupyterHub

JupyterHub offers an interactive way of using our HPC resource, which most researchers use. Besides the typical usage of Jupyter notebooks, starting an RStudio server or a virtual desktop is possible. Further, setting up your own container with the specific software that fulfills your requirements is possible. This way, sharing your setup with project partners is also easy. This way, it is possible to run your workloads with CPU and GPU support.

Metashape

Orthomosaic Generation with Metashape is a workflow we developed for the project ForestCare.

SynForest

SynForest is a tool that generates realistic large-scale point clouds of forests by simulating the lidar scanning process from a stationary or moving platform and capturing the shape and structure of realistic tree models.

Data Pools

Data pools are a simple way to share your data on our HPC system with others. Further, data pools offer access to large data sets that are useful for different user groups over a longer period of time.

Workshops

The GWDG Academy offers workshops on different topics relevant to forest science on HPC. Most of them handle technical topics without examples from the field of forest science. The following are some workshops that include forest science workflows.

Deep learning with GPU cores

As different AI methods are applied in forest science, the workshop Deep learning with GPU cores provides a starting point for your AI workload on our HPC system. The workshop covers tree species classification of segmented point clouds with PointNet. In the section Deep Learning Containers in HPC a container is set up, that provides all required software for the workflow and can be used interactively on JupyterHub or via SLURM on our HPC cluster.

Performance of AI Workloads

Based on the workshop “Deep learning with GPU cores,” we introduce different tools and approaches to optimizing job performance. This can help reduce your waiting time and the energy consumption of your analysis.

Forests in HPC

Forests in HPC introduces different options for using the HPC system for forest applications. Here, we focus on lidar data analysis with R (lidR) and the same deep learning workflow that is used in the workshop Deep learning with GPU cores.

Life Science and Bioinformatics

Description

Life science benefits greatly from the application of computational resources and tools, from large-scale genomic analysis and complex molecular simulations, to sophisticated statistical analysis and image post-processing. We provide access to many bioinformatics and life science tools, the compute power to utilize them, as well as the technical expertise to help researchers access our resources and services.

Access

You can request access to these services via SCC, NHR or KISSKI as explained here.

Applications

The software stack available in our compute clusters is comprehensive, and consequently rather complex. You can find (many) more details in the software stacks page, and the many subpages of list of modules. In this page we will briefly mention how you can install your own programs if need be, and also highlight the more relevant domain programs (some of them with their own, more in-depth pages).

Installing your own programs

Users do not have root permissions, which can make installing your own software a bit more complicated than in your own computer. But still, there are a number of possibilities to achieve your own software stack:

• Compiling from source: This is usually the hardest approach, but should work in most cases. You will have to take care of many dependencies on your own. Most Linux-based programs will follow a configure, make, make install loop, so if you have done this once for a program you can expect a similar procedure for future programs. The only note here is that at the configure step, you will have to define an installation directory that is somewhere in your user space.
• pip: For Python packages, pip is usually the way to go. Make sure the pip you are using matches the python compiler you are actually using (which python and which pip should point to similar locations). To make absolutely sure, you can also use python -m pip install instead of raw pip. Sometimes adding the --user option is necessary to tell pip to install packages to a local folder. Finally, there can sometimes be incompatibilities between packages when using pip, and Python’s module system can be very complex, so we recommend using conda (see next point) or its variants.
• conda: conda is a package and environment manager, particularly popular in the Bioinformatics domain, that makes handling complex environments and dependency stacks easy and transferable. We have a dedicated page with more information about python package management. Do be aware we discourage using conda init when setting up the package manager for the first time, please check our documentation for more details on how to properly use this tool.
• spack: Spack is another package manager, similar to Conda. It can be more complex to use, but at the same time much more powerful. We provide information on how users can make use of Spack for installing their own software in its dedicated page.
• Containers: Containers are a good option for particularly problematic programs with complex requirements, are very transferable, and are increasingly provided as an option directly from developers. We offer access to Apptainer on our cluster, and some of our tools are provided as containers already.
• Jupyter-Hub: Our Jupyter-Hub offers the possibility to run your own custom containers in interactive mode, which can also be reused as containers for batch jobs. This includes Jupyter and Python containers, RStudio (you can also install extra modules on top of the base Python and RStudio containers without having to modify the container itself), and even containers for graphical applications on the HPC-Desktops.
• Cluster-wide installation: In some cases, we can install software for the whole cluster in our module system, depending on how useful and widely used we believe it would be. Contact us at our usual support addresses if you require this.

Genomics

• ABySS: De-novo, parallel, paired-end sequence assembler that is designed for short reads.
• bedtools: Tools for a wide-range of genomics analysis tasks. - BLAST-plus: Basic Local Alignment Search Tool.
• Bowtie: Ultrafast, memory-efficient short read aligner for short DNA sequences (reads) from next-gen sequencers.
• bwa: Burrow-Wheeler Aligner for pairwise alignment between DNA sequences.
• DIAMOND: Sequence aligner for protein and translated DNA searches, designed for high performance analysis of big sequence data.
• GATK: Genome Analysis Toolkit Variant Discovery in High-Throughput Sequencing Data.
• HISAT2: fast and sensitive alignment program for mapping next-generation sequencing reads (whole-genome, transcriptome, and exome sequencing data) against the general human population (as well as against a single reference genome).
• IGV: The Integrative Genomics Viewer is a high-performance visualization tool for interactive exploration of large, integrated genomic datasets.
• IQ-TREE: Efficient software for phylogenomic inference.
• JELLYFISH: A tool for fast, memory-efficient counting of k-mers in DNA.
• Kraken2: System for assigning taxonomic labels to short DNA sequences, usually obtained through metagenomic studies.
• MaSuRCA: Whole genome assembly software. It combines the efficiency of the de Bruijn graph and Overlap-Layout-Consensus (OLC) approaches.
• MetaHipMer (MHM): De-novo metagenome short-read assembler.
• MUSCLE: Widely-used software for making multiple alignments of biological sequences.
• OpenBLAS: An optimized BLAS library.
• RepeatMasker: Screen DNA sequences for interspersed repeats and low complexity DNA sequences.
• RepeatModeler: De-novo repeat family identification and modeling package.
• revbayes: Bayesian phylogenetic inference using probabilistic graphical models and an interpreted language.
• Salmon: Tool for quantifying the expression of transcripts using RNA-seq data.
• samtools: Provides various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating alignments in a per-position format.

Additionally, we offer some tools as containers (which as such might not appear explicitly in the module system), among them: agat, cutadapt, deeptools, homer, ipyrad, macs3, meme, minimap2, multiqc, Trinity, umi_tools. See the tools in containers page for more information.

Molecular Simulations

• Classic molecular simulations: LAMMPS, GROMACS, NAMD.
• Various ab-initio codes: Gaussian, Turbomole, Quantum Espresso, CP2K, CPMD, Psi4, VASP (own license required).
• Alphafold and other protein folding programs: See for example the Protein-AI service.

Imaging

• Freesurfer: an open source software suite for processing and analyzing brain MRI images.
• RELION: (REgularised LIkelihood OptimisatioN, pronounce rely-on) Empirical Bayesian approach to refinement of (multiple) 3D reconstructions or 2D class averages in electron cryo-microscopy (cryo-EM).

Workflow Tools

• Snakemake.
• Nextflow.

Other

• R, Matlab, Python, Octave, and many other programming languages.
• Environment and package handling tools:
  • Conda, Miniforge, uv: see Python.
  • For other software: Spack.
• Neuromorphic computing tools.

Support

For ways to contact us, see support.

More

Mathematics

Description

Numerics related to mathematics can be done with many multi-purpose languages for which we offer support. In addition, we also offer many software packages with more specialized use cases.

Access

You can get access to our NHR resourced by following the NHR application process.

Applications

Current available modules

• BLAS (Basic Linear Algebra Subprograms) — A routine that provides standard building blocks for performing basic vector, matrix operations and the development of high-quality linear algebra software.
• FFTW3 — A C-subroutine library for computing discrete Fourier transforms in one or more dimensions, of arbitrary input size, and of both real and complex data.
• Git – A fast, scalable, distributed revision control system.
• GSL (GNU Scientific Library) — A numerical library for C and C++ programmers. The library provides various mathematical routines such as random number generators, special functions, and least-squares fitting.
• Matlab – A universal interactive numerical application system with advanced graphical user interface.
• METIS – A set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill-reducing orderings for sparse matrices.
• MUMPS (MUltifrontal Massively Parallel sparse direct Solver) — A package for solving systems of linear equation based on a multifrontal approach.
• NFFT (Nonequispaced Fast Fourier Transform) — A C subroutine library for computing the Nonequispaced Discrete Fourier Transform (NDFT) and its generalisations in one or more dimensions, of arbitrary input size, and of complex data.
• Octave – A high-level language, primarily intended for numerical computations.
• ParMETIS – An MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, and meshes, and for computing fill-reducing orderings of sparse matrices.
• PETSc – A Portable, Extensible Toolkit for Scientific Computation: widely used parallel numerical software library for partial differential equations and sparse matrix computations.
• R – A language and environment for statistical computing and graphics that provides various statistical and graphical techniques: linear and nonlinear modeling, statistical tests, time series analysis, classification, clustering, etc.
• ScaLAPACK — A library of high-performance linear algebra routines for parallel distributed memory machines. It solves dense and banded linear systems, least squares problems, eigenvalue problems, and singular value problems.
• Scotch — A software package and libraries for sequential and parallel graph partitioning, static mapping, sparse matrix block ordering, and sequential mesh and hypergraph partitioning.

Support

Contact us using nhr-support@gwdg.de.

More

Physics

Description

Numerics for computational physics can be done with many multi-purpose languages for which we offer support. In addition, we also offer software packages with more specialized use cases.

Access

You can get access to our NHR resourced by following the NHR application process.

Applications

Current available modules

• Ansys Suite - The full Ansys academic Multiphysics Campus Solution is available, e.g. Mechanical, CFX, Fluent, LS-Dyna, Electronic, SCADE (but not Lumerical, GRANTA).
• BLAS (Basic Linear Algebra Subprograms) — A routine that provides standard building blocks for performing basic vector, matrix operations and the development of high-quality linear algebra software.
• FFTW3 — A C-subroutine library for computing discrete Fourier transforms in one or more dimensions, of arbitrary input size, and of both real and complex data.
• Foam-extend — The Open Source CFD Toolbox
• Git – A fast, scalable, distributed revision control system.
• GSL (GNU Scientific Library) — A numerical library for C and C++ programmers. The library provides various mathematical routines such as random number generators, special functions, and least-squares fitting.
• Matlab – A universal interactive numerical application system with advanced graphical user interface.
• METIS – A set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill-reducing orderings for sparse matrices.
• MUMPS (MUltifrontal Massively Parallel sparse direct Solver) — A package for solving systems of linear equation based on a multifrontal approach.
• NFFT (Nonequispaced Fast Fourier Transform) — A C subroutine library for computing the Nonequispaced Discrete Fourier Transform (NDFT) and its generalisations in one or more dimensions, of arbitrary input size, and of complex data.
• Octave – A high-level language, primarily intended for numerical computations.
• OpenFOAM — An object-oriented Computational Fluid Dynamics(CFD) toolkit
• ParaView — An interactive data analysis and visualisation tool with 3D rendering capability
• ParMETIS – An MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, and meshes, and for computing fill-reducing orderings of sparse matrices.
• PETSc – A Portable, Extensible Toolkit for Scientific Computation: widely used parallel numerical software library for partial differential equations and sparse matrix computations.
• R – A language and environment for statistical computing and graphics that provides various statistical and graphical techniques: linear and nonlinear modeling, statistical tests, time series analysis, classification, clustering, etc.
• ScaLAPACK — A library of high-performance linear algebra routines for parallel distributed memory machines. It solves dense and banded linear systems, least squares problems, eigenvalue problems, and singular value problems.
• Scotch — A software package and libraries for sequential and parallel graph partitioning, static mapping, sparse matrix block ordering, and sequential mesh and hypergraph partitioning.

Support

Contact our Support addresses.

Quantum Computing

Description

Quantum computing is a cutting-edge field that leverages the principles of quantum mechanics to process information in fundamentally new ways. Unlike classical computers, which use bits (0s and 1s) for computation, quantum computers use qubits, which can exist in a superposition of states. This enables quantum computers to perform certain types of calculations exponentially faster than classical machines. Quantum computing has the potential to revolutionize multiple industries by solving problems that are difficult or impossible for classical computers. Currently, we do not have our own quantum computer and only provide access to the quantum simulators via containers.

Access

You can request access to this service via SCC or NHR resources as explained here.

Applications

Current available simulators

• Qiskit-aer
• Qulacs
• Cirq and Qsim
• QuTip
• Qibo

Support

Contact us using nhr-support@gwdg.de.

More

Multiple programs and multiple data

Using multiple programs on different data within a single job takes a bit ofset up, as you need to tell the MPI starter exactly what to run and where to run it.

Jobscript

Example script hello.slurm for a code with two binaries

• one OpenMP binary hello_omp.bin running on 1 node, 2 MPI tasks per node and 4 OpenMP threads per task,
• one MPI binary hello_mpi.bin running on 2 nodes, 4 MPI tasks per node.

#!/bin/bash
#SBATCH --time=00:10:00
#SBATCH --nodes=3
#SBATCH --partition=medium:test
 
module load impi
export SLURM_CPU_BIND=none
export OMP_NUM_THREADS=4
 
scontrol show hostnames $SLURM_JOB_NODELIST | awk '{if(NR==1) {print $0":2"} else {print $0":4"}}' > machines.txt
mpirun -machine machines.txt -n 2 ./hello_omp.bin : -n 8 ./hello_mpi.bin

GPU Usage

Step-by-Step guide

If you would like an overview guide on this topic, we have a Youtube playlist to set up and run an example deep learning workflow. You can follow along our step-by-step guide. Note: The content of this guide is outdated.

Partitions and Hardware

There are multiple partitions with GPU nodes available. You will only have access to some of them, depending on what type of user account you have. An overview of all nodes with GPUs can be displayed by running the following command from a frontend node:

sinfo -o "%25N  %5c  %10m  %32f  %10G %18P " | grep gpu

For more details, see GPU Partitions.

Which Frontend to Use

Of the different login nodes available, we have two (glogin9 and 10, it is recommended to use the DNS alias glogin-gpu.hpc.gwdg.de) dedicated to run GPU workloads. These are the closest match to the hardware of our GPU nodes, so it is strongly recommended to use them when writing and submitting GPU jobs and especially when compiling software. While it is technically possible to do so from the other login nodes as well, it is not recommended and may cause problems. For example, a lot of GPU-specific software modules are not even available on other login (or compute) nodes.

Getting Access

Nodes can be accessed using the respective partitions. On the shared partitions (see GPU Partitions), you have to specify how many GPUs you need with the -G x option, where x is the number of GPUs you want to access. If you do not use MPI, please use the -c #cores parameter to select the needed CPUs. Note that on non-shared partitions, your jobs will use nodes exclusively as opposed to sharing them with other jobs. Even if you request less than i.e. 4 GPUs, you will still be billed for all GPUs in your reserved nodes! Note that on the partitions where the GPUs are split into slices via MIG, -G x requests x slices. To explicitly request an 80GB GPU please add the option --constraint=80gb to your jobscript.

Example

the following command gives you access to 32 cores and two A100 GPUs:

srun -p grete:shared --pty -n 1 -c 32 -G A100:2 bash

If you want to run multiple concurrent programs, each using one GPU, here is an example:

#!/bin/bash
#SBATCH -p grete
#SBATCH -t 12:00:00
#SBATCH -N 1

srun --exact -n1 -c 16 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 16 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 16 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 16 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
wait

More explanation for the above example can be found here.

Requesting GPUs

Each Grete node consists of 4 A100 GPUs with either 40 GiB or 80GiB of GPU memory (VRAM), or 8 A100 GPUs with 80 GiB of VRAM (only in grete:shared). In these partitions, you can only request whole GPUs.

• On non-shared partitions (e.g. grete, grete-h100, …) you automatically block the whole node, getting all 4 GPUs on each node. This is suited best for very large jobs. There are no nodes with 8 GPUs in this partition.
• Other GPU partitions are shared, you can choose how many GPUs you need. Requesting less than four (or eight) GPUs means more than one job can run on a node simultaneously (particularly useful for job arrays).

It is possible to request less or more system memory than the default. If you wanted 20GiB of RAM, you could use the additional Slurm argument --mem=20G. We recommend to not explicitly request memory or CPU cores at all, in most cases Slurm will assign an appropriate amount, proportional to the number of GPUs you reserved.

The example script below requests two A100 GPUs:

#!/bin/bash

#SBATCH --job-name=train-nn-gpu
#SBATCH -t 05:00:00                  # Estimated time, adapt to your needs
#SBATCH --mail-type=all              # Send mail when job begins and ends

#SBATCH -p grete:shared              # The partition
#SBATCH -G A100:2                    # Request 2 GPUs

module load miniforge3
module load gcc
module load cuda
source activate dl-gpu               # Replace with the name of your miniforge/conda environment

# Print out some info.
echo "Submitting job with sbatch from directory: ${SLURM_SUBMIT_DIR}"
echo "Home directory: ${HOME}"
echo "Working directory: $PWD"
echo "Current node: ${SLURM_NODELIST}"

# For debugging purposes.
python --version
python -m torch.utils.collect_env
nvcc -V

# Run the script:
python -u train.py

Interactive Usage and GPU Slices on grete:interactive

Whole A100 GPUs are powerful, but you might not need all the power. For instance, debugging a script until it runs might require a GPU, but not the full power of an A100. To this end, the grete:interactive partition is provided. The idea is that you can come into the office, log into the cluster, and use this for development and testing. When you are confident your script works, you should submit larger workloads as a “hands-off” batch job, making use of one or more full GPUs (multiples of 4 for grete).

The interactive partition contains A100 GPUs that are split into slices with the NVIDIA Multi-Instance-GPU (MIG) technology. Each GPU slice consists of a computation part (a subset of the streaming multiprocessors) and some memory.

An A100 for example can be split into 7 compute units comprised of 14 streaming multiprocessors (SMs) each. When using MIG, 10 of the 108 SMs are used for management and are not available for compute. For example, we currently split the 80GiB A100s into six 1g.10gb slices and one 1g.20gb slice, which means that each split has one compute unit (“1g”) and 10 GiB or 20 GiB of GPU memory (VRAM). For an interactive node that has 4 such A100 GPUs, this means there are in total 28 slices per node.

The configuration of MIG slices might be subject to change, depending on the load of the cluster and requirements reported to us by our users. Use scontrol show node <nodename> to see the type and number of slices a node currently offers. MIG slices are configured by an administrator and can not be changed by job scripts, you have to pick one of the sizes on offer.

Instead of requesting a whole GPU with e.g. -G A100:n, as you would in the other GPU partitions, you request MIG slices in the same format Nvidia uses, i.e. -G 1g.10gb:n. The :n for the count of slices at the end is optional, if you don’t specify it, you will get one such slice. At the moment, that is also the maximum allowed for grete:interactive. The following interactive Slurm example requests a 1g.10gb slice:

srun --pty -p grete:interactive -G 1g.10gb /bin/bash

Monitoring

Once you submitted a job, you can check its status and where it runs with

squeue --me

In Your Script

Many packages that use GPUs provide a way to check the resources that are available. For example, if you do deep learning using PyTorch, you can always check that the correct number of resources are available with torch.cuda.is_available().

Note that in PyTorch, the command torch.cuda.get_device_name() does not work with Nvidia-MIG splited GPUs and will give you an error.

Using nvitop

To monitor your resource usage on the node itself, you can use nvitop.

1. Check where your job runs with squeue --me
2. Log into the node. For instance, if your job runs on ggpu146, use ssh ggpu146
3. On the node, run

   module load py-nvitop
   nvitop

In this example output, you can see your

• GPU compute usage/utilization (top) as UTL
• GPU memory usage (top) as MEM
• your CPU usage (bottom) with your abbreviated user name.

Software and Libraries

Cuda Libraries

To load CUDA, do

module load gcc/VERSION cuda/VERSION

If you don’t specify the VERSION, the defaults (12.6.2 for cuda) will be used.

Nvidia HPC SDK (Nvidia compiler)

The full Nvidia HPC SDK 23.3 is installed and useable via the modules nvhpc/23.3 nvhpc-byo-compiler/23.3 nvhpc-hpcx/23.3 nvhpc-nompi/23.3.

This SDK includes the Nvidia compiler and the HPC-X OpenMPI and several base libraries for CUDA based GPU acceleration.

Using MPI and other communication libraries

We have several CUDA enabled OpenMPI versions available. We have the HPC-X OpenMPI in the module nvhpc-hpcx/23.3, the regular OpenMPI 3.1.5 from the Nvidia HPC SDK in nvhpc/23.3 and the OpenMPI included in the Nvidia/Mellanox OFED stack, provided by the module openmpi-mofed/4.1.5a1.

Additional the libraries NCCL 12.0, NVSHMEM 12.0 and OpenMPI 4.0.5 are available as part of the Nvidia HPC SDK in /sw/compiler/nvidia/hpc_sdk/Linux_x86_64/23.3/comm_libs/.

Singularity/Apptainer Containers

Apptainer (see Apptainer (formerly Singularity)) supports “injecting” NVIDIA drivers plus the essential libraries and exectuables into running containers. When you run SIF containers, pass the --nv option. An example running the container FOO.sif would be:

#!/bin/bash
#SBATCH -p grete:shared              # the partition
#SBATCH -G A100:1                    # For requesting 1 GPU.
#SBATCH -c 4                         # Requestion 4 CPU cores.

module load apptainer/VERSION
module load gcc/GCCVERSION
module load cuda/CUDAVERSION

apptainer exec --nv --bind /scratch FOO.sif

where VERSION is the desired version of Apptainer, GCCVERSION the necessary prerequisite gcc version for your chosen version of cuda and CUDAVERSION is the specific version of CUDA desired.

For more information, see the Apptainer GPU documentation.

Using Conda/Mamba/Miniforge

Conda (replaced on our clusters by miniforge3) and Mamba are package managers that can make installing and working with various GPU related packages a lot easier. See Python for information on how to set them up.

Training Resources (Step-by-Step)

We have regular courses on GPU usage and deep learning workflows. Our materials are online for self-studying.

• A deep learning example for training a neural network on Grete: https://gitlab-ce.gwdg.de/dmuelle3/deep-learning-with-gpu-cores
• Example GPU jobs are being collected at https://gitlab-ce.gwdg.de/gwdg/hpc-usage-examples/-/tree/main/gpu

Interactive Jobs

What is an interactive job? Why use interactive jobs?

An interactive job requests resources from a partition, and immediately opens a session on the assigned nodes so you can work interactively. This is done usually on specially designated interactive or test partitions, that have low to no wait times (but also usually low maximum resource allocation and short maximum sessions), so your session can start immediately. This can also be attempted on normal partitions, but of course you would have to be present at the terminal when your job actually starts.

There are multiple use cases for interactive jobs:

• Performing trial runs of a program that should not be done on a log-in node. Remember login nodes are in principle just for logging in!
• Testing a new setup, for example a new Conda configuration or a Snakemake workflow, in a realistic node environment. This prevents you from wasting time in a proper partition with waiting times, just for your job to fail due to the wrong packages being loaded.
• Testing a new submission script or SLURM configuration.
• Running heavy installation or compilation jobs or Apptainer container builds.
• Running small jobs that don’t have large resource requirements, thus reducing waiting time.
• Doing quick benchmarks to determine the best partition to run further computations in (e.g. is the code just as fast on Emmy Phase 2 nodes as Emmy Phase 3)
• Testing resource allocation, which can sometimes be tricky particularly for GPUs. Start up a GPU interactive job, and test if your system can see the number and type of GPUs you expected with nvidia-smi. Same for other resources such as CPU count and RAM memory allocation. Do remember interactive partitions usually have low resource maximums or use older hardware, so this testing is not perfect!
• Running the rare interactive-only tools and programs.

How to start an interactive job

To start a (proper) interactive job:

srun -p jupyter --pty -n 1 -c 16 bash

This will block your terminal while the job starts, which should be within a few minutes. If for some reason this is taking too long or it returns a message that the request is denied (due to using the wrong partition or exceeding resource allocations for example), you can break the request with Ctrl-c.

In the above command:

• starts a job in the partition designated after -p
• --pty runs in pseudo terminal mode (critical for interactive shells)
• then the usual SLURM resource allocation options
• finally we ask it to start a session in bash

You should see something like the following after your command, notice how the command line prompt (if you haven’t played around with this) changes after the job starts up and logs you into the node:

u12345@glogin5 ~ $ srun -p standard96:test --pty -n 1 -c 16 bash
srun: job 6892631 queued and waiting for resources
srun: job 6892631 has been allocated resources
u12345@gcn2020 ~ $ 

To stop an interactive session and return to the login node:

exit

Which partitions are interactive?

Any partition can be used interactively if it is empty enough, but some are specialized for it with shorter wait times and thus better suited for interactive jobs. These interactive partitions can change as new partitions are added or retired. Check the list of partitions for the most current information. Partitions whose names match the following are specialized for shorter wait times:

• *:interactive
• *:test which have shorter maximum job times
• jupyter* which is shared with the Jupyter-HPC service and are overprovisioned (e.g. your job may share cores with other jobs)

You can also just look for other partitions with nodes in the idle state (or mixed nodes if doing a job that doens’t require a full node on a shared partition) with sinfo -p PARTITION. For example, if we check the scc-gpu partition:

[scc_agc_test_accounts] u12283@glogin6 ~ $ sinfo -p scc-gpu
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
scc-gpu      up 2-00:00:00      1  inval ggpu194
scc-gpu      up 2-00:00:00      3   mix- ggpu[135,138,237]
scc-gpu      up 2-00:00:00      1   plnd ggpu145
scc-gpu      up 2-00:00:00      1  down* ggpu150
scc-gpu      up 2-00:00:00      1   comp ggpu199
scc-gpu      up 2-00:00:00      1  drain ggpu152
scc-gpu      up 2-00:00:00      1   resv ggpu140
scc-gpu      up 2-00:00:00     11    mix ggpu[139,141,147-149,153-155,195-196,212]
scc-gpu      up 2-00:00:00      6  alloc ggpu[136,142-144,146,211]
scc-gpu      up 2-00:00:00      4   idle ggpu[151,156,197-198]

we can see that there are 4 idle nodes and 11 mixed nodes. This means that an interactive job using a single node should start rather quickly, particularly if it only requires part of a node since then one of the mixed nodes might be able to run it too.

Pseudo-interactive jobs

If you have a job currently running on a given node, you can actually SSH into that node. This can be useful in some cases to debug and check on your program and workflows. For example, you can check on the live GPU load with nvidia-smi or monitor the CPU processes and the host memory allocation with btop. Some of these checks are easier and more informative when performed live rather than using after-job reports such as the job output files or sacct.

u12345@glogin5 ~ $ squeue --me
  JOBID    PARTITION         NAME     USER  ACCOUNT     STATE       TIME  NODES NODELIST(REASON)
6892631   standard96         bash   u12345  myaccount   RUNNING     11:33     1 gcn2020
u12345@glogin5 ~ $ ssh gcn2020
u12345@gcn2020 ~ $

If you try this on a node you don’t currently have a job in it will fail, since its resources have not been allocated to your user!

GPU Interactive Jobs

See: GPU Usage.

Interactive Jobs with Internet Access

See: Internet access within jobs.

Job arrays

Job arrays are the preferred way to submit many similar jobs, for instance, if you need to run the same program on several input files, or run it repeatedly with different settings or parameters. This type of parallelism is usually called “embarrassingly parallel” or trivially parallel jobs.

Arrays are created with the -a start-finish sbatch parameter. E.g. sbatch -a 0-19 will create 20 jobs indexed from 0 to 19. There are different ways to index the arrays, which are described below.

Job Array Indexing, Stepsize and more

Slurm supports a number of ways to set up the indexing in job arrays.

• Range: -a 0-5
• Multiple values: -a 1,5,12
• Step size: -a 0-5:2 (same as -a 0,2,4)
• Combined: -a 0-5:2,20 (same as -a 0,2,4,20)

Additionally, you can limit the number of simultaneously running jobs with the %x parameter in there:

• -a 0-11%4 only four jobs at once
• -a 0-11%1 run all jobs sequentially
• -a 0-5:2,20%2 everything combined. Run IDs 0,2,4,20, but only two at a time.

You can read everything on array indexing in the sbatch man page.

Slurm Array Environment Variables

The behavior of your applications inside array jobs can be tied to Slurm environment variables, e.g. to tell the program which part of the array they should process. These variables will have different values for each job in the array. Probably the most commonly used of these environment variables is $SLURM_ARRAY_TASK_ID, which holds the index of the current job in the array. Other useful variables are:

SLURM_ARRAY_TASK_COUNT: Total number of tasks in a array.
SLURM_ARRAY_TASK_MAX: Job array’s maximum ID (index) number.
SLURM_ARRAY_TASK_MIN: Job array’s minimum ID (index) number.
SLURM_ARRAY_TASK_STEP: Job array’s index step size.
SLURM_ARRAY_JOB_ID: Job array’s master job ID number.

Example job array

The most simple example for using a job array is running a loop in parallel.

#!/bin/bash
#SBATCH -p medium
#SBATCH -t 10:00
#SBATCH -n 1
#SBATCH -c 4

module load python
for i in {1..100}; do
  python myprogram.py $i
done

#!/bin/bash
#SBATCH -p medium
#SBATCH -t 10:00
#SBATCH -n 1
#SBATCH -c 4
#SBATCH -a 1-100

module load python
python myprogram.py $SLURM_ARRAY_TASK_ID

The loop in the first example runs on the same node and in serial. More efficiently, the job array in the second tab unrolls the loop and if enough resources are available, runs all of the 100 jobs in parallel.

Example job array running over files

This is an example of a job array, creates a job for every file ending in .inp in the current working directory:

#!/bin/bash
#SBATCH -p medium
#SBATCH -t 01:00
#SBATCH -a 0-X
# insert X as the number of .inp files you have -1 (since bash arrays start counting from 0)
# ls *.inp | wc -l
 
#for safety reasons
shopt -s nullglob
#create a bash array with all files
arr=(./*.inp)
 
#put your command here. This just runs the fictional "big_computation" program with one of the files as input
./big_computation ${arr[$SLURM_ARRAY_TASK_ID]}

In this case, you have to get the number of files beforehand (fill in the X). You can also do that automatically by removing the #SBATCH -a line and adding that information on the command line when submitting the job:

sbatch -a 0-$(($(ls ./*.inp | wc -l)-1)) jobarray.sh

The part in parentheses just uses ls to output all .inp files, counts them with wc and subtracts 1, since bash arrays start counting at 0.

Job runtimes and QoS

Very short jobs

In case you have jobs that are shorter than 2 hours, there is the option to set the QoS 2h, which will drastically increase your priority. This option can be set via --qos=2h or in a batch script via '#SBATCH --qos=2h.

Please note that you need to specify a time limit that is at most 2 hours --time=2:00:00 as well.

Job Walltime

The maximum runtime is set per partition and can be seen either on the system with sinfo or here. There is no minimum walltime (we cannot stop your jobs from finishing, obviously), but a walltime of at least 1 hour is strongly recommended. Our system is optimized for high performance, not high throughput. A large amount of smaller, shorter jobs induces a lot of overhead as each job has a prolog, setting up the enviroment, an epilog for cleaning up and bookkeeping which can put a lot of load on the scheduler. The occasional short job is fine, but if you submit larger amounts of jobs that finish (or crash) quickly, we might have to intervene and temporarily suspend your account. If you have lots of smaller workloads, please consider combining them into a single job that runs for at least 1 hour. A tool often recommended to help with issues like this (among other useful features) is Jobber.

Jobs that need more than 12 hours of runtime

The default runtime limit on most partitions is 12 hours. On systems where your compute time is limited (you can check with the sbalance command), we will only refund jobs that run up to 12 hours, beyond that you are running the application at your own risk. But you can use the --time option in your sbatch script to request up to 48 hours of runtime.

Jobs that need more than 48 hours of runtime

Most compute partitions have a maximum wall time of 48 hours. Under exceptional circumstances, it is possible to get a time extension for individual jobs (past 48 hours) by writing a ticket during normal business hours. To apply for an extension, please write a support request, containing the Job ID, the username, the project and the reason why the extension is necessary. Alternatively - under even more exceptional circumstances and also via mail request, including username, project ID and reason - permanent access to Slurm Quality-Of-Service (QoS) levels can be granted, which permit a longer runtime for jobs (up to 96 hours) but have additional restrictions regarding job size (e.g. number of nodes).

Please make sure you have checked out all of the following suggestions before you request access to the 96h qos:

1. Can you increase the parallelization of your application to reduce the runtime?
2. Can you use checkpointing to periodically write out the intermediate state of your calculation and restart from that in a subsequent job?
3. Are you running the code on the fastest available nodes?
4. Is it possible to use GPU acceleration to speed up your calculation?
5. Are you using the latest software revision (in contrast to e.g. using hlrn-tmod or rev/xx.yy)?

Dependent & Restartable Jobs - How to pass the wall time limit

If your simulation is restartable, it might be handy to automatically trigger a follow-up job. Simply provide the ID of the previous job as an additional sbatch argument:

# submit a first job, extract job id
jobid=$(sbatch --parsable job1.sbatch)
 
# submit a second job with dependency: starts only if the previous job terminates successfully)
jobid=$(sbatch --parsable --dependency=afterok:$jobid job2.sbatch)
 
# submit a third job with dependency: starts only if the previous job terminates successfully)
jobid=$(sbatch --parsable --dependency=afterok:$jobid job3.sbatch)

Multiple concurrent programs on a single node

Using srun to create multiple jobs steps

You can use srun to start multiple job steps concurrently on a single node, e.g. if your job is not big enough to fill a whole node. There are a few details to follow:

• By default, the srun command gets exclusive access to all resources of the job allocation and uses all tasks.
  • Therefore, you need to limit srun to only use part of the allocation.
  • This includes implicitly granted resources, i.e. memory and GPUs.
  • The –exact flag is needed.
  • If running non-mpi programs, use the -c option to denote the number of cores, each process should have access to.
• srun waits for the program to finish, so you need to start concurrent processes in the background.
• Good default memory per cpu values (without hyperthreading) are usually are:

Examples

#!/bin/bash
#SBATCH -p standard96
#SBATCH -t 06:00:00
#SBATCH -N 1
 
srun --exact -n1 -c 10 --mem-per-cpu 3770M  ./program1 &
srun --exact -n1 -c 80 --mem-per-cpu 3770M  ./program2 &
srun --exact -n1 -c 6 --mem-per-cpu 3770M  ./program3 &
wait

#!/bin/bash
#SBATCH -p gpu
#SBATCH -t 12:00:00
#SBATCH -N 1
 
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
srun --exact -n1 -c 10 -G1 --mem-per-cpu 19075M  ./single-gpu-program &
wait

Using the Linux parallel command to run a large number of tasks

If you have to run many nearly identical but small tasks (single-core, little memory) you can try to use the Linux parallel command. To use this approach you first need to write a bash-shell script, e.g. task.sh, which executes a single task. As an example we will use the following script:

#!/bin/bash
 
# parallel task
TASK_ID=$1
PARAMETER=$((10+RANDOM%10))    # determine some parameter unique for this task
                               # often this will depend on the TASK_ID
 
echo -n "Task $TASK_ID: sleeping for $PARAMETER seconds ... "
sleep $PARAMETER
echo "done"

This script is simply defining a variable PARAMETER which then used as the input for the actual command, which is sleep in this case. The script also takes one input parameter, which can be interpreted as the TASK_ID and could also be used for determining the PARAMETER. If we make the script executable and run it as follows, we get:

$ chmod u+x task.sh
$ ./task.sh 4
Task 4: sleeping for 11 seconds ... done

To now run this task this task 100 times with different TASK_IDs we can write the following job script:3

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/bin/bash
 
#SBATCH --partition medium40:test        # adjust partition as needed
#SBATCH --nodes 1                        # more than 1 node can be used
#SBATCH --tasks-per-node 40              # one task per CPU core, adjust for partition
 
# set memory available per core
MEM_PER_CORE=4525    # must be set to value that corresponds with partition
 
# Define srun arguments:
srun="srun -n1 -N1 --exclusive --mem-per-cpu $MEM_PER_CORE"
# --exclusive     ensures srun uses distinct CPUs for each job step
# -N1 -n1         allocates a single core to each task
 
# Define parallel arguments:
parallel="parallel -N 1 --delay .2 -j $SLURM_NTASKS --joblog parallel_job.log"
# -N                number of argument you want to pass to task script
# -j                number of parallel tasks (determined from resources provided by Slurm)
# --delay .2        prevents overloading the controlling node on short jobs
# --resume          add if needed to use joblog to continue an interrupted run 
#                   (job resubmitted)
# --joblog          creates a log-file, required for resuming
 
# Run the tasks in parallel
$parallel "$srun ./task.sh {1}" ::: {1..100}
# task.sh          executable(!) script with the task to complete, may depend on some input 
#                  parameter  
# ::: {a..b}       range of parameters, alternatively $(seq 100) should also work  
# {1}              parameter from range is passed here, multiple parameters can be used 
#                  with additional {i}, e.g. {2} {3} (refer to parallel documentation)  

The script use parallel in line 25 to run task.sh 100 times with a parameter taken from the range {1..100}. Because each task is started with srun a separate job step is created and the options used with srun (see line 12) the task is using only a single core. This simple example can be adjusted as needed by modifying the script task.sh and the job script parallel_job.sh. You can adjust the requested resources, for example, you can use more than a single node. Note that depending on the number of tasks you may have to split your job into several to keep the total time needed short enough. Once the setup is done, you can simply submit the job:

$ sbatch parallel_job.sh

Looping over two arrays

You can use parallel to loop over multiple arrays. The –xapply option controls, if all permuatations are used or not:

$ parallel --xapply echo {1} {2} ::: 1 2 3 ::: a b c
1 a
2 b
3 c
$ parallel echo {1} {2} ::: 1 2 3 ::: a b c
1 a
1 b
1 c
2 a
2 b
2 c
3 a
3 b
3 c

Resource monitoring and Reports

When debugging and optimizing your application, it is important to know what is actually happening on the compute node. Most information has to be collected during the runtime of the job, but a few key metrics are still available after a job has finished.

During Runtime

sstat

During a job’s runtime, you can use the sstat command to get live information about its performance, regarding CPU, Tasks, Nodes, Resident Set Size (RSS) and Virtual Memory (VM) for each step.

To use, call sstat -a <jobid> to list all jobsteps of a single job. Due to amount of fields displayed, we recommend appending a | less -S to the command to enable side-scrolling, or to reduce the amount of displayed fields with the -o flag. (Important note: When forgetting the -a flag, sstat may display an empty dataset when the job did not start explicit jobsteps).

You can find all available fields and their meaning, as well as further information about the command in its man page (man sstat) or on its website https://slurm.schedmd.com/sstat.html.

SSH

While a job is running, you can also use ssh to get onto the node(s) allocated to your job to view its performance directly. Use squeue --me to see which nodes your job is running on. Once logged in to the compute node, you can take a look at the resource usage with standard Linux commands, such as htop, ps or free. Please keep in mind that most commands will show ALL resources of the node, not just those allocated to you.

After the Job finished / Reports

sacct

sacct is a Slurm tool to query the Slurm database for information about jobs. The returned information range from simple status information (current runstate, submission time, nodes, etc.) to gathered performance metrics.

To use, call sacct -j <jobid>, which will display a small subset of the available job information (Please keep in mind that database operations are async, thus recently started jobs might not yet be in the database and thus are not available in sacct).

To get more than the basic job information immediately available, one can use the --long flag, which will print many more fields per job. Due to the large number of fields and thus hard to read output, we recommend appending a | less -S to the output to enable side-scrolling in the output. Further fields can be manually selected using the --format flag, where the command sacct --helpformat lists all available fields.

As a form of special field, the flag --batch-script will print the slurm script of the job if the job was submitted with sbatch and the flag --env-vars will print the list of environment variables of a job, usefull for debugging. These two flags cannot be combined with others, expect the -j flag.

You can find all available fields and flags, as well as further information about the command in its man page (man sacct) or on its website https://slurm.schedmd.com/sacct.html.

reportseff

To get resource efficiency information about your job after it finished, you can use the tool reportseff. This tools queries Slurm to get your allocated resources and compares it to the actually used resources (as reported by Slurm). You can get all information reportseff uses to create reports by manually using sacct, it just collates them in a nice fashion.

This can give you a great overview about your usage: Did I make use of all my cores and all my memory? Was my time limit too long?

Usage example:

# Display your recent jobs
gwdu101:121 14:00:00 ~ > module load py-reportseff
gwdu101:121 14:00:05 ~ > reportseff -u $USER
     JobID    State       Elapsed  TimeEff   CPUEff   MemEff 
 
  12671730  COMPLETED    00:00:01   0.0%      ---      0.0%  
  12671731  COMPLETED    00:00:00   0.0%      ---      0.0%  
  12701482  CANCELLED    00:04:20   7.2%     49.6%     0.0%
 
 
# Give specific Job ID:
gwdu102:29 14:07:17 ~ > reportseff 12701482
     JobID    State       Elapsed  TimeEff   CPUEff   MemEff 
  12701482  CANCELLED    00:04:20   7.2%     49.6%     0.0%  

As you can see in the example, the job only took 4:20 minutes out of 1h allocated time, resulting in a TimeEfficiency of 7.2%. Only half the allocated cores (two allocated and only one used) and basically none of the allocated memory were used. For the next similar job, we should reduce the time limit, request one core less and definitely request less memory.

Self-Resubmitting jobs

Do you find yourself in the situation that your jobs need more time than allowed? Do you regularly write tickets to lengthen your job times or wait longer because of using QOS=long? Self-Resubmitting jobs might be a solution for you!

The requirement is that the program you are running is able to or can be updated so that is produces checkpoints and is able to restart from any checkpoint after a forced stop. Many programs already have check pointing options, like Gromacs or OpenFoam. Turn those on and update your batch script to resubmit itself to continue running.

Two different types of examples are shown below that run a python script called big-computation.py, which in the first example is creating a checkpoint every 10 minutes and can restart from the last checkpoint file. The second example continuously writes to an output file which can be copied into a checkpoint, which in turn can be used at the restart of the computation.

#!/bin/bash
#SBATCH -p standard96
#SBATCH -t 12:00:00
#SBATCH -N 1
#SBATCH -c 4
#SBATCH -o outoput_%j.txt
#SBATCH -e error_%j.txt

if [ ! -f "finished" ] ; then
	sbatch --dependency=afterany:$SLURM_JOBID resub_job.sh
else
	exit 0
fi

# big-computation creates automatic checkpoints
# and automatically starts from the most recent 
# checkpoint
srun ./big-computation.py --checkpoint-time=10min --checkpoint-file=checkpoint.dat input.dat

# If big-computation.py is not canceled due to time out
# write the finished file to stop the loop.
touch finished

#!/bin/bash
#SBATCH -p standard96
#SBATCH -t 12:00:00
#SBATCH -N 1
#SBATCH -c 4
#SBATCH -o outoput_%j.txt
#SBATCH -e error_%j.txt
#SBATCH --signal=B:10@300
# Send signal 10 (SIG_USR1) 5min before time limit

trap SIGUSR1 'pkill -f big-computation.py -15 ; cp output.dat checkpoint.dat ; exit 1'

JOBID=`sbatch --dependency=afternotok:$SLURM_JOBID resub_job_trap.sh`

if [ -f "checkpoint.dat" ] ; then
	INPUT=checkpoint.dat
else
	INPUT=input.dat
fi

# big-computation creates automatic checkpoints
# and automatically starts from the most recent 
# checkpoint
srun ./big-computation.py $INPUT

scancel $JOBID
exit 0

Both scripts rely on the --dependency flag from the sbatch command. The first example runs until the scheduler kills the process. The next job starts after this one has finished, either by being killed or finishing. Only when the computation did successfully finish, a file called finished will be written, breaking the loop.

The first script will start and wait for one more job once the computation has finished because it always checks for the finished file.

The second script traps a signal to stop the computation. Observe the option #SBATCH --signal=B:10@300, which tells the scheduler to send the signal SIGUSER1(10) 5 minutes before the job is killed due to the time limit. The command trap SIGUSER1 captures this signal, stops the program with the SIGINT(15) signal, copies the output file into a checkpoint from which the computation can resume, and exits the script with code 1 (an error code). The resubmitting sbatch command handles the dependency to start the script only after the last job exited with an error code. Once started, the program either uses the input file or the last checkpoint file as the input, and the script only exits successfully if the computations finishes.

The second script will keep the last submitted job in the queue even though the last job finished successfully. It will be a pending job with reason DependencyNeverSatisfied and this job needs to be canceled manually using the scancel command. Therefore, the job script saves the submitted jobid and cancels it directly once the program exits normally.

Program specific examples

These examples take the ideas explored above and apply it to specific programs.

#!/bin/bash
#SBATCH -p standard96
#SBATCH -t 12:00:00
#SBATCH -N 1
#SBATCH -c 4
#SBATCH -o outoput_%j.txt
#SBATCH -e error_%j.txt

JOBID=`sbatch --dependency=afternotok:$SLURM_JOBID resub_job_gromacs.sh`

module load impi
module load gromacs

mpirun -np 1 gmx_mpi mdrun -s input.tpr -cpi checkpoint.cpt

scancel $JOBID
exit 0

CPU Partitions

Nodes in these partitons provide many CPU cores for parallelizing calculations.

Islands

The islands with a brief overview of their hardware are listed below.

Partitions

The NHR partitions following the naming scheme sizeCORES[suffix] where size indicates the amount of RAM (medium, standard, large, or huge), CORES indicates the number of cores, and suffix is only included to differentiate partitions with the same size and CORES. SCC, KISSKI, REACT, etc. partitions do not follow this scheme. The partitions are listed in the table below by which users can use them and island without hardware details. See Types of User Accounts to determine which kind of user you are. Note that some users are members of multiple classifications (e.g. all CIDBN/FG/SOE users are also SCC users).

The hardware for the different nodes in each partition are listed in the table below. Note that some partitions are heterogeneous, having nodes with different hardware. Additionally, many nodes are in more than one partition.

The CPUs

For partitions that have heterogeneous hardware, you can give Slurm options to request the particular hardware you want. For CPUs, you can specify the kind of CPU you want by passing a -C/--constraint option to slurm to get the CPUs you want. Use -C ssd or --constraint=ssd to request a node with a local SSD. If you need a particularly large amount of memory, please use the --mem option to request an appropriate amount (per node). See Slurm for more information.

The CPUs, the options to request them, and some of their properties are give in the table below.

Hardware Totals

The total nodes, cores, and RAM for each island are given in the table below.

GPU Partitions

Nodes in these partitons provide GPUs for parallelizing calculations. See GPU Usage for more details on how to use GPU partitions, particularly those where GPUs are split into MiG slices.

Islands

The islands with a brief overview of their hardware are listed below.

Partitions

The partitions are listed in the table below by which users can use them, without hardware details. See Types of User Accounts to determine which kind of user you are. Note that some users are members of multiple classifications (e.g. all CIDBN/FG/SOE users are also SCC users).

The hardware for the different nodes in each partition are listed in the table below. Note that some partitions are heterogeneous, having nodes with different hardware. Additionally, many nodes are in more than one partition.

How to pick the right partition for your job

If you have access to multiple partitions, it can be important to choose one that fits your use case. As a rule of thumb, if you need (or can scale your job to run on) mutliple GPUs, and you are not using a shared partition, make sure to always use a multiple of 4 GPUs, as you will be billed for the whole node regardless of a lower number of GPUs you requested via -G. For jobs that need less than 4 GPUs, use a shared partition and make sure to not request more than your fair share of RAM (see the note above). If you need to get your to start quickly, i.e. for testing if your scripts work or interactive tweaking of hyperparameters, use an interactive partition (most users have access to grete:interactive).

The CPUs and GPUs

For partitions that have heterogeneous hardware, you can give Slurm options to request the particular hardware you want. For CPUs, you can specify the kind of CPU you want by passing a -C/--constraint option to slurm to get the CPUs you want. For GPUs, you can specify the name of the GPU when you pass the -G/--gpus option (or --gpus-per-task) and larger VRAM or a minimum in CUDA Compute Capability using a -C/--constraint option. See Slurm and GPU Usage for more information.

The GPUs, the options to request them, and some of their properties are given in the table below.

The CPUs, the options to request them, and some of their properties are give in the table below.

Hardware Totals

The total nodes, cores, GPUs, RAM, and VRAM for each island are given in the table below.

Spack FAQs

Frequently Asked Questions (FAQs) about Spack

1. What is Spack?

Question: What is Spack?

Answer: Spack is a package manager designed for high-performance computing (HPC) applications. It simplifies the process of building and installing multiple versions of software on various platforms.

2. How do I install Spack?

Question: How do I install Spack?

Answer: To install Spack, you can clone the Spack repository from GitHub and add it to your environment. Use the following commands:

git clone https://github.com/spack/spack.git
. spack/share/spack/setup-env.sh

However on our systems, you don’t have to install Spack, it is already provided as a module, you can just load it with:

module load spack

module load spack-user

module load spack

3. How do I search for packages in Spack?

Question: How do I search for packages in Spack?

Answer: You can search for packages in Spack using the spack list command followed by the package name or a part of the package name. For example:

spack list <package-name>

4. How do I install a package using Spack?

Question: How do I install a package using Spack?

Answer: To install a package, use the spack install command followed by the package name. For example, to install HDF5, you would run:

spack install hdf5

5. How do I load a package installed with Spack?

Question: How do I load a package installed with Spack?

Answer: You can load a package into your environment using the spack load command. For example:

spack load hdf5

6. How do I create a custom package in Spack?

Question: How do I create a custom package in Spack?

Answer: To create a custom package, you need to write a package recipe in Python. This involves defining the package’s metadata and build instructions in a file within the var/spack/repos/builtin/packages directory. Detailed instructions can be found in the Spack documentation.

7. How do I update Spack to the latest version?

Question: How do I update Spack to the latest version?

Answer: You can update Spack by pulling the latest changes from the GitHub repository:

cd $SPACK_ROOT
git pull

Then, you may need to run Spack commands to update its database and environment.

8. How do I report a bug or request a feature in Spack?

Question: How do I report a bug or request a feature in Spack?

Answer: You can report bugs or request features by opening an issue on the Spack GitHub repository. Be sure to provide detailed information about the bug or feature request.

9. Where can I find more information about Spack?

Question: Where can I find more information about Spack?

Answer: More information about Spack, including comprehensive documentation and tutorials, can be found on the official Spack website and the Spack documentation.

Cluster Storage Map

Not every data store is available everywhere on the cluster, nor is the filesystem performance equal for all nodes. The differences depend on which island a node is a member of and whether it is a login or a compute node. General CPU node islands are called “Emmy Phase X” where X indicates the hardware generation (1, 2, 3, …). General GPU node islands are called “Grete Phase X” where X indicates the hardware generation (1, 2, 3, …). Other islands exist for specific institutions/groups or historical reasons (e.g. SCC Legacy). Which systems can be accessed and their relative performance of the link between each group/island of nodes (logins and partitions) and each data store is shown in the diagram below

There are a few important points to note:

• Portal SCC users have access to some CPU and GPU nodes in the Grete Phase 2 & 3 and Emmy Phase 3 islands (the scc-cpu and scc-gpu partitions) in addition to the SCC Legacy island.
• Legacy SCC users only have access to the SCC Legacy island and have their HOME directories on the GWDG Unix HOME filesystem.
• Each island (SCC Legacy, Emmy Phase 2, Grete & Emmy Phase 3) has a separate SCRATCH/WORK data store connected via a very fast network, with slower connections to the other ones.
• ARCHIVE/PERM data stores are only accessible from login nodes.
• Temporary Storage for Slurm jobs is always allocated in RAM, on fast local SSDs, and the fastest SCRATCH/WORK available for the node.
• The CIDBN, FG, and SOE islands are not shown in the diagram above but have access to the same storage systems with the same relative performance as the SCC Legacy, though some also have their own dedicated SCRATCH/WORK.

Quota

To see the quotas for any data stores your current user has access to, use the command show-quota (for more details, see the dedicated page below).

Fundamentals

Quotas are used to limit how much space and how many files users and projects can have in each data store. This prevents things like a runaway or buggy job using up the filesystem’s entire capacity rendering it unusable for other users and projects on the cluster, as well as preventing steady accumulation of unused files. Data stores have two different kinds of quotas which may be set:

Block

• Quota limiting how much space can be used (i.e. how many blocks on the filesystem).
• Note that because filesystems store files in blocks, the usage rounds up to the nearest block in many cases (e.g. a 1 byte file often takes up 4 KiB of space).

Inode

• Quota limiting how many files, directories, and links can be used.
• In most filesystems, every file, directory, and link takes up at least 1 inode.
• Large directories and large files can sometimes take up more than 1 inode.

For each kind of quota, filesystems have two different kinds of limits – hard and soft limits where the soft limit, if present, is less than or equal to the hard limit.

Hard limit

When one is above the hard limit, the filesystem prevents consuming more blocks/inodes and may restrict what changes can be done to existing blocks/inodes until one drops below the soft limit.

Soft limit

When one is above the soft limit but below the hard limit, a timer starts giving the user/project a certain grace time to drop below the soft limit again. Before the grace time expires, the user/project is not restricted so long as they stay below the hard limit. If the grace time expires without dropping below the soft limit, the filesystem restricts the user/project the same as exceeding the hard limit until the usage drops below the soft limit. Often a warning is printed to stderr when the soft limit is first exceeded, but there is no message via E-Mail warning about reaching the soft limit.

Essentially, the soft limit is the value one must stay below but one may briefly exceed without problems so long as one does not exceed the hard limit.

Quota Pages

The different quota sub-topics are covered in the pages below

• Checking Usage And Quotas
• Fixing Quota Issues
• Increasing Quota

Workspaces

Workspaces allow users to request temporary storage with an expiration date on the high performance filesystems. This increases the efficiency and overall performance of the HPC system, as files that are no longer required for compute jobs do not accumulate and fill up these filesystems.

The typical workflow is that the workspaces commands are used to request and create a directory on the appropriate filesystem before a new chain of compute jobs is launched. After the job chain has finished, the results should be copied to a more permanent location in a data store shared with other project members.

Each project and project user can have multiple workspaces per data store, each with their own expiration dates. After the expiration date, the expired workspace and the data within it are archived in an inaccessible location and eventually deleted.

The concept of workspaces has become quite popular and is applied in a large number of HPC centers around the world. We use the well known HPC Workspace tooling from Holger Berger to manage workspaces.

Workspaces are meant for active data and are configured for high aggregate bandwidth (across all files and nodes at once) at the expense of robustness. The bandwidth for single files varies based on the underlying filesystem and how they are stored, please check the filesystem pages for more details.

Common workflows are to use workspaces for temporary data, or store data in a compact form in a Project or COLD data store and then copy and/or decompress it to a higher performance workspace to be used for a short period of time. The workspaces have a generous quota for the whole project (the finite lifetimes of workspaces help protect against running out of space as well).

The data stores for workspaces available to each kind of project are:

Workspace Basics

The six basic commands to handle workspaces and manage their lifecycles are:

All six commands have help messages accessible by COMMAND -h and man pages accessible by man COMMAND.

Workspaces are created with the requested expiration time (each data store has a maximum allowed value). The default expiration time if none is requested is 1 day. Workspaces can have their expiration extended a limited number of times. After a workspace expires, it is released. Released workspaces can be restored for a limited grace period, after which the data is permanently deleted. Note that released workspaces still count against your filesystem quota during the grace period. All workspace tools use the -F <name> option to control which data store to operate on, where the default depends on the kind of project. The various limits, default data store for each kind of project, as well as which cluster islands each data store is meant to be used from and their purpose/specialty are:

Managing Workspaces

Allocating

Workspaces are created via

ws_allocate [OPTIONS] WORKSPACE_NAME DURATION

The duration is given in days and workspace names are limited to ASCII letters, numbers, dashes, dots, and underscores. The most important options (run ws_allocate -h to see the full list) are:

  -F [ --filesystem ] arg   filesystem
  -r [ --reminder ] arg     reminder to be sent n days before expiration
  -m [ --mailaddress ] arg  mailaddress to send reminder to
  -g [ --group ]            group workspace
  -G [ --groupname ] arg    groupname
  -c [ --comment ] arg      comment

Use --reminder <days> --mailaddress <email> to be emailed a reminder the specified number of days before the workspace expires. Use --group --groupname <group> to make the workspace readable and writable by the members of the specified group, however this only works for members of the group that are also members of the same project. Members of other projects (than the username you used to create the workspace) cannot access it, even if you have a common POSIX group and use the group option. Thus, usually the only value that makes sense is the group HPC_<project>, which can be conveniently generated via "HPC_${PROJECT_NAME}" in the shell. If you run ws_allocate for a workspace that already exists, it just prints its path to stdout, which can be used if you forgot the path (you can also use ws_list).

To create a workspace named MayData on ceph-ssd with a lifetime of 6 days which emails a reminder 2 days before expiration, you could run:

~ $ ws_allocate -F ceph-ssd -r 2 -m myemail@example.com MayData 6
Info: creating workspace.
/mnt/ceph-ssd/workspaces/ws/nhr_internal_ws_test/u17588-MayData
remaining extensions  : 2
remaining time in days: 6

The path to the workspace is printed to stdout while additional information is printed to stderr. This makes it easy to get the path and save it as an environment variable:

~ $ WS_PATH=$(ws_allocate -F ceph-ssd -r 2 -m myemail@example.com MayData 6)
Info: creating workspace.
remaining extensions  : 2
remaining time in days: 6
~ $ echo $WS_PATH
/mnt/ceph-ssd/workspaces/ws/nhr_internal_ws_test/u17588-MayData

You can set defaults for the duration as well as the --reminder, --mailaddress, and --groupname options by creating a YAML config file at $HOME/.ws_user.conf formatted like this:

duration: 15
groupname: HPC_foo
reminder: 3
mail: myemail@example.com

Listing

Use the ws_list command to list your workspaces, the workspaces made available to you by other users in your project, and the available datastores. Use the -l option to see the available data stores for your username on the particular node you are currently using:

~ $ ws_list -l
available filesystems:
ceph-ssd
lustre-rzg
DONT_USE (default)

Note that the special unusable location DONT_USE is always listed as the default even if the default for the kind of project your username is in is available.

Running ws_list by itself lists your workspaces that can be accessed from the node (not all data stores are available on all nodes). Add the -g option to additionally list the ones made available to you by other users. If you run ws_list -g after creating the workspace in the previous example, you would get:

~ $ ws_list -g
id: MayData
     workspace directory  : /mnt/ceph-ssd/workspaces/ws/nhr_internal_ws_test/u17588-MayData
     remaining time       : 5 days 23 hours
     creation time        : Thu Jun  5 15:01:14 2025
     expiration date      : Wed Jun 25 15:01:14 2025
     filesystem name      : ceph-ssd
     available extensions : 2

Extending

The expiration time of a workspace can be extended with ws_extend up to the maximum time allowed for an allocation on the chosen data store. It is even possible to reduce the expiration time by requesting a value lower than the remaining duration. The number of times a workspace on a particular data store can be extended is also limited, to two times on our cluster. Workspaces are extended by running:

ws_extend -F DATA_STORE WORKSPACE_NAME DURATION

Don’t forget to specify the data store with -F <data-store>. For example, to extend the workspace allocated in the previous example to 20 days, run:

~ $ ws_extend -F ceph-ssd MayData 20
Info: extending workspace.
Info: reused mail address example@example.com
/mnt/ceph-ssd/workspaces/ws/nhr_internal_ws_test/u17588-MayData
remaining extensions  : 1
remaining time in days: 20

Releasing

A workspace can be released before its expiration time by running:

ws_release -F DATA_STORE [OPTIONS] WORKSPACE_NAME

The most important option here is --delete-data which causes the workspace’s data to be deleted immediately (remember, the data stores for workspaces have NEITHER backups NOR snapshots, so the data is lost forever). Otherwise, the workspace will be set aside and remain restorable for the duration of the grace period of the respective data store.

Restoring

A released or expired workspace can be restored within the grace period using the ws_restore command. Use ws_restore -l to list your restorable workspaces and to get their full IDs. If the previously created example workspace was released, you would get:

~ $ ws_restore -l
ceph-ssd:
u17588-MayData-1749129222
        unavailable since Thu Jun  5 15:13:42 2025
lustre-rzg:
DONT_USE:

Note that the full ID of a restorable workspace includes your username, the workspace name, and the unix timestamp from when it was released. In order to restore a workspace, you must first have another workspace available on the same data store to restore it into. Then, you would call the command like this:

ws_restore -F DATA_STORE WORKSPACE_ID_TO_RESTORE DESTINATION_WORKSPACE

and it will ask you to type back a set of randomly generated characters before restoring (restoration is interactive and is not meant to be scripted). The workspace being restored is placed as a subdirectory in the destination workspace with its ID. Using the previous example, one could create a new workspace MayDataRestored and restore the workspace to it via:

~ $ WS_DIR=$(ws_allocate -F ceph-ssd MayDataRestored 30)
Info: creating workspace.
remaining extensions  : 2
remaining time in days: 30
~ $ ws_restore -F ceph-ssd u17588-MayData-1749129222 MayDataRestored
to verify that you are human, please type 'tafutewisu': tafutewisu
you are human
Info: restore successful, database entry removed.
~ $ ls $WS_DIR
u17588-MayData-1749129222

Managing Links

Keeping track of the paths to each workspace can be difficult sometimes. You can use the ws_register DIR command to setup symbolic links (symlinks) to all of your workspaces in the directory DIR. After doing that, each of your workspaces has a symlink <dir>/<datastore>/<username>-<workspacename>.

~ $ mkdir ws-links
~ $ ws_register ws-links
keeping link  ws-links/ceph-ssd/u17588-MayDataRestored
~ $ ls -lh ws-links/
total 0
drwxr-x--- 2 u17588 GWDG 4.0K Jun  5 15:47 DONT_USE
drwxr-x--- 2 u17588 GWDG 4.0K Jun  5 15:47 ceph-ssd
drwxr-x--- 2 u17588 GWDG 4.0K Jun  5 15:47 lustre-rzg
~ $ ls -lh ws-links/*
ws-links/DONT_USE:
total 0

ws-links/ceph-ssd:
total 0
lrwxrwxrwx 1 u17588 GWDG 71 Jun  5 15:47 u17588-MayDataRestored -> /mnt/ceph-ssd/workspaces/ws/nhr_internal_ws_test/u17588-MayDataRestored

ws-links/lustre-rzg:
total 0

Temporary Storage

These can be used to reduce the IO load on slower data stores and therefore improve job performance and reduce the impact on other users. A common workflow is

1. Copy files that must be read multiple times, particularly in random order, into the temporary storage.
2. Do computations, keeping the most intense operations in the temporary storage (e.g. output files that have to be written over many times or in a random order).
3. Copy the output files that need to be kept from the temporary storage to some data store.

In this workflow, the temporary storage is used as a staging area for the high intensity IO operations while the other storage locations get low intensity ones (e.g. reading a file beginning to end once in large chunks).

Temporary Storage in a Job

Each batch job has several temporary storages available, which are listed in the table below along with their characteristics. A temporary directory is created in each available one whose path is put into an environmental variable for convenient use. These directories are cleaned up (deleted) when the job ends (no need to manually clean up).

The environmental variable TMPDIR will be assigned to one of these. You can override that decision by setting it yourself to the value of one of them. For example, to use the local SSD, you would run the following in Bash:

export TMPDIR=$LOCAL_TMPDIR

The local temporary stores have the best performance because all operations stay within the node and don’t have to go over the network. But that inherently means that the other nodes can’t access them. The global temporary stores are accessible by all nodes in the same job, but this comes at the expense of performance. It is possible to use more than one of these in the same job (e.g. use local ones for the highest IO intensity files local to the node and a global one for things that must be shared between nodes). We recommend that you choose the local ones when you can (files that don’t have to be shared between nodes and aren’t too large).

Shared Memory

The fastest temporary storage is local shared memory (stored under /dev/shm) which is a filesystem in RAM. It has the smallest latency and bandwidth can exceed 10 GiB/s in many cases. But it is also the smallest.

The size of the files you create in it count against the memory requested by your job. So make sure to request enough memory in your job for your shared memory utilization (e.g. if you plan on using 10 GiB in it, you need to increase the -m MEMORY amount you request to Slurm by 10 GiB).

Optimizing IO Performance

It is important to remember that the data stores are shared by with other users and bad IO patterns can hurt not only the performance of your jobs but also that of other users. A general recommendation for distributed network filesystems is to keep the number of file metadata operations (opening, closing, stat-ing, truncating, etc.) and checks for file existence or changes as low as possible. These operations often become a bottleneck for the IO of your job, and if bad enough can reduce the performance for other users. For example, if jobs request hundreds of thousands metadata operations like open, close, and stat; this can cause a “slow” filesystem (unresponsiveness) for everyone even when the metadata is stored on SSDs.

Therefore, we provide here some general advice to avoid making the data stores unresponsiveness:

• Use the local temporary storage of the nodes when possible in jobs (see Temporary Storage for more information).
• Write intermediate results and checkpoints as seldom as possible.
• Try to write/read larger data volumes (>1 MiB) and reduce the number of files concurrently open.
• For inter-process communication use proper protocols (e.g. MPI) instead of using files to communicate between processes.
• If you want to control your jobs externally, consider using POSIX signals instead of using files frequently opened/read/closed by your program. You can send signals to jobs by scancel --signal=SIGNALNAME JOBID
• Use MPI-IO to coordinate your I/O instead of each MPI task doing individual POSIX I/O (HDF5 and netCDF may help you with this).
• Instead of using resursive chmod/chown/chgrp, please use as combination of find (note that Lustre has its own optimized lfs find) and xargs. For example, lfs find /path/to/folder | xargs chgrp PROJECTGROUP creates less stress than chgrp -R PROJECTGROUP /path/to/folder and is much faster.

Analysis of Metadata Operations

An existing application can be investigated with respect to metadata operations. Let us assume an example job script for the parallel application myexample.bin with 16 MPI tasks.

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=01:00:00
#SBATCH --partition=standard96

srun ./myexample.bin

The linux command strace can be used to trace IO operations by prepending it to the call to run another program. Then, strace traces that program and creates two files per process (MPI task) with the results. For this example, 32 trace files are created. Large MPI jobs can create a huge number of trace files, e.g. a 128 node job with 128 x 96 MPI tasks created 24576 files. That is why we strongly recommend to reduce the MPI tasks when doing performance analysis as far as possible.

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=01:00:00
#SBATCH --partition=standard96

srun strace -ff -t -o trace -e open,openat ./myexample.bin

Analysing one trace file shows all file open activity of one process (MPI task).

> ls -l trace.*
-rw-r----- 1 bzfbml bzfbml 21741 Mar 10 13:10 trace.445215
...
> wc -l trace.445215
258 trace.445215
> cat trace.445215
13:10:37 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
13:10:37 open("/lib64/libfabric.so.1", O_RDONLY|O_CLOEXEC) = 3
...
13:10:38 open("/scratch/usr/bzfbml/mpiio_zxyblock.dat", O_RDWR) = 8

For the interpretation of the trace file you need to differentiate between the calls originating from your code and the ones that are independent of it (e.g. every shared library your code uses and their shared libraries and so on has to be opened at least once). The example code myexample.bin creates only one file with the name mpiio_zxyblock.dat. 258 open statements in the trace file include only one open from the application which indicates a very desirable meta data activity.

Known Issues

Some codes have well known issues:

• OpenFOAM: always set runTimeModifiable false and fileHandler collated with a sensible value for purgeWrite and writeInterval (see the OpenFOAM page)
• NAMD: special considerations during replica-exchange runs (see the NAMD page)

If you have questions or you are unsure regarding your individual scenario, please get in contact with your consultant or start a support request.

Filesystem Specific Tips

Lustre

Some good best practices for using the Lustre SCRATCH/WORK data stores can be found at https://www.nas.nasa.gov/hecc/support/kb/lustre-best-practices_226.html

Technical Description

Each cluster provides several different storage systems that can be placed into the following rough categories:

[1]: These numbers are generalized figures and only show the soft limits.
[2]: Being phased out in favor of Workspaces. See SCRATCH/WORK for the phase out schedule.
[3]: These are the numbers for the SSD scratch and the HDD scratch, you can find more information here.

These are shared parallel filesystems accessible from many nodes, rather than node local storage (which the nodes also have). These filesystems also have quotas on the use of storage space (block quota) and inodes (every file and directory has an entry in the directory tree using one or more inodes).

See the pages below (also can be found in the nav bar to the left) for information about the different parts of the storage systems and how to use them.

• Data Stores
• Ceph
• Lustre
• Tape Archive
• Vast
• Project Map
• Requestable Storage