Data Migration Guide

While having a separate username for each project has some upsides such as separate data store quotas, never having to worry about submitting jobs with the wrong Slurm account, etc.; a major downside is that sometimes files must be copied or moved between usernames. Common scenarios are:

• Copying scripts or configuration files in your HOME directory that took effort to create (e.g. .bashrc, .config/emacs/init.el, etc.)
• Moving files from your legacy HLRN/SCC user to a new project-specific username
• Moving files from your username of an expired project to the new username of a successor project

Using a common POSIX group (recommended)

If possible, granting your other usernames access via a common POSIX group is the simplest and most widely supported way to share or move files.

To see what groups your current user is a member of, you can use the groups command (or alternatively, id which shows your current username, user id and all groups). To be able to access a file, assuming it is not world-readable or writable, you either need to own it, or be a member of the group that owns the file, given it has the respective read permission and every parent directory on the path to the file has read and execute permissions set.

HPC_u_<academicid> groups

For every HPC Project Portal user, or rather every AcademicID that has HPC users belonging to it, there is a group called HPC_u_<academicid>. For example, John Doe is a member of two projects and thus, has two project specific usernames u12345 and u56789, he will have a group called HPC_u_jdoe with 3 members, the two u-users plus his primary user jdoe, which is only relevant in case he is also a legacy SCC user.

He can now grant access to a directory owned by u12345 to his other username u56789 by changing the group that owns it and giving the group read, execute and optionally write permissions by running:

chgrp -R HPC_u_jdoe /path/to/directory
chmod -R g+rwX /path/to/directory

Assuming every parent directory on the path to directory can also be accessed by u56789. He will then be able to access everything under that directory from both u12345 and u56789, copy (or move) files from there to somewhere else, etc.

If John wants to keep using the shared directory to create new files from either username, he could also set the SGID-bit (recommended), so any newly created files will also be owned by the correct group automatically:

find /path/to/directory -type d -exec chmod g+s {} \;

What if there is no common POSIX group? What about project directories?

You might want to share data between users that have no POSIX groups in common (or the ones they do have in common also contain many other users). For example when migrating from a legacy HLRN account to a Portal account, these are technically two separate AcademicIDs and will not share a common HPC_u_ group by default. But your legacy HLRN account can be added to the HPC_u_ group of your new account by our support team, please write a ticket and don’t forget to clearly state both your legacy username and your new HPC_u_<academicid> group name.

Similarly, your new project specific usernames can be added as members to your legacy HLRN projects’ POSIX group by our admins.

Tips and advanced commands

If you have a very large amount of files/directories and the commands above take a long time to complete, here are some tips to speed it up.

1. For large numbers of directories, set the SGID-bit with this more advanced version of the above command:

find /path/to/directory -type d \! -perm /g+s -print0 | xargs -0rn 200 chmod g+s

2. For a large number of files, some of which may already belong to the correct group or have group r/w/x permissions, changing the group and setting permissions can be sped up by running:

find /path/to/directory \! -group HPC_u_jdoe -print0 | xargs -0rn 200 chgrp HPC_u_jdoe
find /path/to/directory \! -perm /g+rw -print0 | xargs -0rn 200 chmod g+rwX

3. Use a terminal multiplexer to let the commands run over night.

4. Use the correct login node to run your commands. Accessing filesystems for a specific cluster island, if possible from login nodes dedicated to other islands, may be a lot slower than accessing them from the correct login nodes. For scratch-emmy/lustre-emmy-*, use glogin[3-8]. For scratch-grete/lustre-grete, use glogin[9-13]. For scratch-scc, use gwdu[101-102].

Using ACLs

A more complex way to move or copy your data is via ACLs (Access Control Lists). These can be more powerful than regular POSIX permissions, but are not supported on some file systems, are not immediately visible and can easily lead to confusion or mistakes. ACLs should be avoided unless you actually need more fine-grained control that can’t be achieved by regular POSIX permissions.

Let SRCUSERNAME and DESTUSERNAME be your usernames of where the files come from and who they are going to respectively, SRCDIR and DESTDIR be their data store directories (e.g. /scratch-emmy/usr/SRCUSERNAME and /mnt/lustre-emmy-hdd/usr/DESTUSERNAME), and DIR_TO_MIGRATE to be the name of the directory to be be migrated. You can copy data or move data. Copying can be done entirely on your own, but is very heavy on the IO for large data. Moving is more efficient as long as it is on the same filesystem (as expensive as copying data otherwise) but requires a support request for the last step.

Copy Data

Copying data is suitable when the data is at most medium size (both in space and in number of files/directories). But for large and very large sizes, you should instead move data if possible.

First, login to a login node with the source username (e.g. ssh SRCUSERNAME@glogin.hpc.gwdg.de) and do the following:

# Make it so the destination username can even reach the directory
setfacl -m u:DESTUSERNAME:x SRCDIR

# Make the whole directory readable for the destination username, and then make
# it possible for the destination username to enter directories (x permission)
# which has to be done with find
setfacl -R -m u:DESTUSERNAME:r SRCDIR/DIR_TO_MIGRATE
find $SRCDIR/DIR_TO_MIGRATE -type d -print0 \
    | xargs --null --no-run-if-empty setfacl -m u:DESTUSERNAME:rx

Then, login to a login node with the destination username (e.g. ssh DESTUSERNAME@glogin.hpc.gwdg.de) and do the following to copy the data.

cp -r SRCDIR/DIR_TO_MIGRATE DESTDIR/

Then, back with the source username, remove the created ACLs with

setfacl -x u:DESTUSERNAME SRCDIR
setfacl -R -x u:DESTUSERNAME SRCDIR/DIR_TO_MIGRATE

Move Data

This is the best approach if the data is very large as it is quite cheap from an IO perspective as long as the source and destination are on the same filesystem. It also works when there is no intention of using the files/directories by the source username anymore. Unfortunately, this method does require starting a support request for the last step.

First, login to a login node with the destination username (e.g. ssh DESTUSERNAME@glogin.hpc.gwdg.de) and do following to setup the move (giving SRCUSERNAME the permissions to enter and create/delete files):

setfacl -m u:SRCUSERNAME:wx DESTDIR

Then, login to a login node with the source username (e.g. ssh SRCUSERNAME@glogin.hpc.gwdg.de) and do the following to move the data:

mv SRCDIR/DIR_TO_MIGRATE DESTDIR/

Then, back with the destination username, remove the ACL that let SRCUSERNAME do the operation with

setfacl -x u:SRCUSERNAME DESTDIR

At this point, the data is moved but it has the wrong owner and group (those of SRCUSERNAME). Start a support request (see Getting Started for the correct email address) requesting that the directory’s ownership be changed from your SRCUSERNAME to your DESTUSERNAME.