Terminal multiplexer

Some commands you can’t or might not want to run in a batch job can take a long time to complete. This is especially true for interactive commands, but also for larger file transfers or calculating the size of a large directory. If your ssh connection to the cluster is interrupted during that process, your session on the server is terminated and the commands fail half-way. If you do not want to or can’t keep the terminal session open, because you are connecting over an unstable network or from a battery-powered device, there is the option to use either tmux or screen. Both are terminal multiplexers with different capabilities.

The basic idea is to log onto a server, such as an HPC login node, via ssh and start a multiplexer session there. Running ssh within a multiplexer session on your laptop for example would not help, since your session on the remote server will still be killed when the connection is interrupted. Having a multiplexer session on the server allows you to log out and even turn off your computer without affecting your server-side session. Your long-running command can continue running on the server without the need for an open connection, and you can reconnect at a later point and resume your multiplexer session.

In order to reconnect, make sure you are on the same server/login node that you started the session on. For more information, see the paragraph Why can’t I resume my session?.

tmux

A new tmux session can be started with the command

tmux new -s my_session

The optional -s my_session can be used to give your session a name so you can reconnect more easily later. This will become especially helpful if you have multiple sessions on the same server.

After starting this session, you can now run any commands and work normally. Disconnecting or leaving the session without ending it is called “detaching”. In tmux, that is done by pressing a so-called “prefix key” (or rather a key combination), which is Ctrl + B by default, followed by another key to perform various actions. Detaching is done by pressing the prefix combination followed by D. See the tmux manpage for more information and other possible actions.

You can now close the ssh connection or the terminal, the detached tmux session will keep running in the background. If your connection fails or your local computer crashes, is turned off unexpectedly, etc. your tmux session will be detached automatically and will remain unaffected.

To reattach, once you are connected back to the same server as before, run

tmux attach -t my_session

The parameter -t my_session is optional, if left out tmux will attach to the first available session. A list of currently running sessions can be shown with tmux ls.

To end a session, exit the shell inside of it by running exit or use

tmux kill-session -t my_session

from outside the session.

screen

Screen operates similar to tmux when it comes to handling sessions, but the commands are a little different. To start a screen session named “my_session” for example, run:

screen -S my_session

Detaching from this session is done by pressing Ctrl + A, releasing and then pressing D.

A list of running screen sessions can be shown with screen -ls.

Reattach by running

screen -r my_session

If there is only one session running, just screen -r will also work, otherwise it will also show you the list of sessions.

Similarly, exiting a session can be done by using the exit command in the session or pressing Ctrl + A, followed by K.

Why can’t I resume my session?

In order to reconnect to your session started on a specific server, you need to be on the same server/login node. The sessions are running locally and are not shared between nodes like your directories and files on networked filesystems are, for instance. Therefore, you need to remember the server’s name.

If you are on the wrong server, or your session has ended for some reason, you’ll get messages like no sessions, can't find session my_session or There is no screen to be resumed.

As an example, you connected to a random login node using ssh glogin.hpc.gwdg.de (technically a so called DNS-round-robin), landed on glogin6, started a tmux session and later got disconnected due to a timeout or network interruption. When you connect to the generic DNS name “glogin.hpc.gwdg.de” again, you will likely end up on a different login node than before, e.g. glogin4, and will be unable to resume your session.

Refer to the table on the login nodes and examples page for an overview.

In order to avoid this issue, remember the hostname of the server you started the session on. For example, if your command prompt looks like this:

[my_project] u12345@glogin8 ~ $

you are on glogin8. Later, you can ssh directly to the same as before node by running

ssh glogin8.hpc.gwdg.de

and should have no problems resuming your session.

If that still does not work, the login node may have been rebooted in the meantime, ending your session. Check your emails if there was an announcement or type the command uptime. This does not happen often, but is not avoidable sometimes.

Session sharing with another user

Terminal multiplexers not only allow attaching and detaching from your own sessions, you can also attach to another user’s session if they created a socket file and gave you permission. This can be useful for collaboration, teaching or help with troubleshooting.

Here is how you can setup your own shareable tmux session and give permission to other users to access it.

1. Prepare a temporary directory to place the socket file and give the other person(s) access to it.

The socket file acts as an entry point to all sessions started with -S pointing to it and facilitates the communication between the different tmux clients. The person you want to share your session with will need to be able to read and write to this file.

We recommend using your shared POSIX group HPC_project_id if you want to share a session with another member of your HPC project. For this example though, we will focus on the lowest common denominator that will work for every combination of (two or more) usernames, from different projects, etc. That means we will add an ACL (access control list) to the temporary directory and socket file.

Please read Data Sharing for more information on this topic.

export DIR=$(mktemp -d)
setfacl -m u:u56789:rx $DIR

This creates a temporary directory to which, by default, only you have access. The second command adds an ACL to it that allows your colleague’s username (u56789 in this example, replace it with their actual username your colleague told you) to read its contents and enter (=execute) it. Repeat this command for every additional person you want to share the session with.

2. Start a new tmux session and specify the path to a socket file with -S. The file should not exist before this step and will be created automatically.

tmux -S $DIR/tmux_socket

3. Add an ACL to the socket file that allows u56789 (again, replace this with the actual username) to read and write it.

setfacl -m u:u56789:rw $DIR/tmux_socket

4. To make sure, print the hostname of the login node you are using and path to your socket file and send them to your colleague.

$ hostname
glogin7
$ echo $DIR/tmux_socket
/tmp/tmp.tij8dqc9CU/tmux_socket

5. Your colleague needs to login to the same login node and attach their tmux session to yours via the socket file. If they don’t want to mess with their ssh client configuration, but have configured it to land on a random login node each time, they can also just run ssh glogin7 for example, from another login-node, to start a new ssh session on glogin7 within their existing ssh session.

tmux -S /tmp/tmp.tij8dqc9CU/tmux_socket attach

6. You should now see something like

 |__   __| |  | |  ____|  / ____|/ ____/ ____|                                  │········
    | |  | |__| | |__    | (___ | |   | |                                       │········
    | |  |  __  |  __|    \___ \| |   | |                                       │········
    | |  | |  | | |____   ____) | |___| |____                                   │········
    |_|  |_|  |_|______| |_____/ \_____\_____|                                  │········
                                                                                │········
 Documentation:  https://docs.hpc.gwdg.de                                       │········
 Support:        hpc-support@gwdg.de                                            │········
                                                                                │········
PARTITION    NODES (BUSY/IDLE)     LOGIN NODES                                  │········
medium             95 /    0     login-mdc.hpc.gwdg.de                          │········
scc-cpu            49 /    0     glogin-p3.hpc.gwdg.de                          │········
Slurm load last updated 41.356842 seconds ago                                   │········
Your current login node is part of glogin-p2                                    │········
u12345@glogin7 ~ $ setfacl -m u:u56789:rw $DIR/tmux_socket                      │········
                                                                                │········
u12345@glogin7 ~ $ hostname                                                     │········
glogin7                                                                         │········
u12345@glogin7 ~ $ echo $DIR/tmux_socket                                        │········
/tmp/tmp.tij8dqc9CU/tmux_socket                                                 │········
u12345@glogin7 ~ $                                                              │········
────────────────────────────────────────────────────────────────────────────────┘········
·························································································
·························································································
·························································································
·························································································
·······················································(size 80x21 from a smaller client)
[0] 0:bash*                                                     "glogin7" 09:34 27-Jan-26

Both of you will now see the same output (or at least the upper left section of it). Any key your colleague presses will behave the same as if you pressed it and vice-versa. Both of you can detach from the session individually and reattach with the command from step 5.

7. You can end or kill the session, even if your colleague is still connected, by detaching with your configured tmux “prefix key” (Ctrl+B by default), followed by D and running

tmux -S $DIR/tmux_socket kill-session

In case you started multiple sessions, use kill-server instead, to kill all sessions at once.

8. Don’t forget to clean up by deleting the tempdir and socket file after you are done.

rm -r $DIR