Data Sharing

This page describes how you can collaboratively work on data within the HPC system and make it available to other users or to the outside world.

When running the show-quota command, you will see all data stores your current user has access to, divided into “User” and “Project” stores. Every member of your HPC Project Portal project can access the same project stores you see listed there. You can configure access for other project members to files owned by you via POSIX group permissions. If you are not familiar with the POSIX permission model, please take a look at our introductory course to learn the basics.

Please read Managing Permissions for general advice and tips on how to best manage your files.

Methods for sharing data within the cluster

Here is the list of methods you can utilize to make your data available to other users of the GWDG HPC systems. We recommend that you consider each method in the order they are listed here.

Applying for a project

If you are planning to closely collaborate with people from across multiple working groups for an extended period of time or invite external collaborators for a specific research topic (that should not have access to all your working group’s data), consider applying for a dedicated project. An HPC project will include a shared project directory and allows for dynamically inviting users via the HPC Project Portal. In order to apply as an SCC user, see Applying for SCC projects, otherwise contact our support.

Using a common POSIX group

You can make a directory (or individual files) available to other users by changing its group to one that has you and the other users as members. If you are members of the same project, in the Project Portal you will see ID of HPC project: my_project. Your common POSIX group would then be named HPC_my_project.

Other members of your project already have access to the same project data stores as you. They may also be configured so newly created files and subdirectories inherit the correct group from the parent directory, so you will only need to worry about changing the group permissions with chmod. Most of the time, what you would want to do is add or remove write permission by running chmod -Rh g+w <directory> or the same command with g-w respectively.

If you want to share data with people outside of your project, you will have to use your home directory, a user data store, as they will not have access to your project data stores and you can only give access to directories you own. Remember that in order to access a file or directory, they need to be able to enter (not necessarily read) all parent directories as well. The only directory a regular user owns, that is located directly under a publicly accessible parent, is their home directory. Since your home directory has limited storage quota, this method is only suitable for smaller amounts of data. In order to share larger datasets, see Requesting shared workspaces below.

chgrp <group> $HOME
chmod g=x $HOME
chgrp -Rh <group> <subdirectory>
chmod -Rh g+rX <subdirectory>

Giving a group execute (but not read) permission like this will also allow members to access any file or subdirectory that has other permissions, if they know or can guess the name.

See Managing Permissions for more information and details on how to use chgrp and chmod.

Tip

If you want to share data with your own different usernames, use the group HPC_u_<academicuser> (where academicuser is the username of your AcademicID). Please see the Data Migration Guide for more details.

In order to see what groups your current username and others have available, use id -nG <username>. To see all members of a group, use getent group <groupname>. Please make sure to only share data with groups that do not have many unintended members! Especially, do NOT use a group like HPC_all! Some groups (especially “primary” groups like UMIN, GNOI or GWDG) don’t show any members, but actually have a very large number of members. Those should almost never be used. If there is no fitting group available, please apply for one (as described in the next paragraph).

Applying for a shared group

If a dedicated project is not feasible, and there is no good common POSIX group you already have, it is possible to apply for one by contacting our support. Please provide a good reason why you need a POSIX group, a unique group name and a list of usernames you want to be members of the new group. It is possible and recommended to set up a group that will automatically contain all users of one or more HPC projects, even ones added to one of these projects at a later point in time.

Requesting shared workspaces

Workspaces are the preferred way to get non-permanent storage on fast filesystems for active (“hot”) data used in jobs. They can also be used to collaborate with other projects on research data.

Every member of a project can create workspaces in the project’s designated workspace data stores. By default, a workspace can only be accessed by the person who created it, but there is an option (-g group) to allow access for the specified POSIX group. In the standard configuration of the project data stores, only members of your own project can access the directory containing the workspaces, so even if you gave a different group than your project group access to the workspace itself, members of that group (if they are not also members of your project) would not be able to “get through” to the workspace. This can be changed, i.e. a shared group containing members of your project and the project you want to collaborate with can be created and given access to one of your project’s workspace data stores, so everyone will be able to access workspaces created with -g your_shared_group. If you are the PI or a delegate and want to enable that for your project, please contact our support.

Using a hidden directory

This is more of a temporary measure, and less secure than other methods, but quick and easy to do. The data will be available to all users that know the path to it. You should send the full path only to people you want to give access.

Warning

This method will leave all files and directories with non-empty other permissions under your chosen parent directory, especially those with more predictable names, open to all users of the HPC cluster!

Make sure you understand the basics of the POSIX permission model before attempting it!

While the top-level home or project directories do not have other permissions set by default, many files and subdirectories will likely have. Make sure to unset those before, for example:

chmod -R o= /mnt/vast-nhr/home/jdoe/u12345

Depending on the owning group, it may or may not be advisable to include g in the above command. Read this section to the end (especially the last info box below) for more context.

You have been warned!

Remember that in order to access a file or directory, one needs to be able to access all its parent directories as well. For directories, the read permission decides whether you can list its contents, the write permission decides if you can create or delete files in it, and the execute permission decides whether you can cd into it (or access files and subdirectories if their respective permissions allow for it). The trick is to make one of the parent directories executable only, but not readable.

Info

In this example, we will use the home directory, but you can apply the steps similarly to any other data store. Make sure to use the full canonical path (“real path”) to your directory to avoid confusion!

[nhr_ni_test] u12345@glogin4 ~ $ pwd
/user/jdoe/u12345
[nhr_ni_test] u12345@glogin4 ~ $ realpath .
/mnt/vast-nhr/home/jdoe/u12345

As you can see, the real path is different from the apparent path.

First, create a directory with a random name:

SHAREDIR=$(mktemp -p /mnt/vast-nhr/home/jdoe/u12345 -d share.XXXXXXXX)

This will create a directory with a random name in /mnt/vast-nhr/home/jdoe/u12345 and save the path in the variable SHAREDIR. You can now place the files you want to share in that directory. (Use tab-completion to avoid having to remember the random name or do e.g. cp some_file $SHAREDIR/)

Next, you need to set permissions to the directory:

chmod -R go+rX $SHAREDIR

And make sure the parent directory is not readable, only executable:

chmod go=x $SHAREDIR/..

(The above is equivalent to chmod go=x /mnt/vast-nhr/home/jdoe/u12345 in this example, but works regardless of where you chose to create the “hidden” directory.)

Info

We are changing group as well as other permissions here. This assumes your parent directory is owned by your user’s primary group, which is the case for most home directories by default. These primary groups are most likely very large groups (including everyone from your institution), so chances are high that some of the users you would like to share data with are members of the same primary group, but not all of them. Group permissions take precedence over other permissions, so just setting those permissions for others could exclude users that share your primary group.

If you have changed the owning group of your directory to your HPC_u_<academicuser> group or you placed the hidden directory under a project directory, please remove the g from the last two commands. For example: chmod o=x $SHAREDIR/..

In such cases, leave the group permissions untouched or feel free to set them however you prefer.

Your “hidden” directory is now ready, you can send the path to any other users you want to share the data with. To print the path of the shared directory, run:

echo $SHAREDIR

Users who know the path can cd into it and copy the files to their own directories. When the share is no longer needed, don’t forget to unset execute permission of the parent directory to restrict access to any other subdirectories or files again.

chmod go= /mnt/vast-nhr/home/jdoe/u12345

Using ACLs

ACLs (Access Control Lists) are a more advanced, but also more complex method of defining fine-grained permissions in addition to the traditional POSIX permissions. They work on most filesystems, but not all of them, and are not immediately visible and thus easier to forget or make mistakes with. ACLs should be preferred over the “hidden” directory method above when you want to share a directory long-term and to a smaller number of people. We still recommend to only use them when other methods (like a common project or POSIX group) are not available.

How to set and manage ACLs on your directories is described in our Data Migration Guide.

Sharing data publicly

Last but not least, here are some options to share data with people that are not users of the GWDG HPC systems.

Using Owncloud

Many AcademicIDs have access to GWDGs instance of Owncloud. It can be used to upload the files you want to share directly from the cluster. See this paragraph for details on how to upload to Owncloud using rclone. After you have done so, you can either create a public link that allows anyone to download a copy, or use the sharing feature of Owncloud to select other Academic Cloud users that will be able to see the files after logging into Owncloud themselves.

Using S3

In order to use S3, you have to apply for an S3-Bucket by writing an email to support@gwdg.de and asking for one that is accessible from the HPC system. You can then share your secret key and private key within your group to give everyone access. In this scenario, access to your data is done via http, and it is reachable not only from the HPC systems, but also the Cloud and Internet (if needed).

You can access your S3-Bucket from a compute node using https://s3.gwdg.de as an endpoint.

In order to work with your S3-Bucket, you could for instance use rclone:

module load rclone
# List content of your Bucket
rclone ls <config-name>:<bucket-name>/<prefix>
# Or Snyc the Content of your $HOME with the Bucket
rclone sync -i $HOME/some/folder <config-name>:<bucket-name>/<prefix>
# Or Snyc the Content of your Bucket with your $HOME
rclone sync -i <config-name>:<bucket-name>/<prefix> $HOME/some/folder

This requires a config file in ~/.config/rclone/rclone.conf with the following content:

[<config-name>]
type = s3
provider = Ceph
env_auth = false
access_key_id = <AccessKey>
secret_access_key = <SecretKey>
region =
endpoint = https://s3.gwdg.de
location_constraint =
acl =
server_side_encryption =
storage_class