Fixing Quota Issues

This page will focus on the most common quota problem: being close to the quota limits or exceeding them. If you have another kind of problem, suspect something is wrong, or have already tried everything listed on this page and definitely need more quota, you can contact our support. NHR users are encouraged to first contact your consultant to discuss your needs and how we can help you, after which you would contact support at nhr-support@gwdg.de.

Determine Where Your Usage Is

The filesystem quotas and show-quota work on whole directories and don’t indicate anything about where in a directory the usage is. Often times, a lot of the usage is old files and directories or caches that are no longer needed and can be cleaned. Use show-quota to get the path/s to each of your (or your project’s) directories and data stores. Then, you can determine where your usage is in those directories: either manually with ls and du, or using a tool with a Terminal User Interface (TUI) frontend like ncdu (see below).

If you are in a directory, you can run

ls -alh

to list all files and directories including hidden ones (the -a option) along with the sizes of all the files in a human readable format (the -l and -h options put together). The -a option is particularly useful in your HOME directory where there are many hidden files and directories. An example of its output would be

[nhr_ni_test] u12345@glogin5 ~ $ ls -lah
total 79M
drwx------    4 u12345 GWDG  4.0K May 27 15:09 .
drwxr-xr-x 4606 root   root  268K May 27 10:50 ..
lrwxrwxrwx    1 u12345 GWDG     4 May 27 15:09 .a -> .baz
drwxr-x---    2 u12345 GWDG  4.0K May 27 15:08 bar
-rw-r-----    1 u12345 GWDG   78M May 27 15:08 .baz
drwxr-x---    2 u12345 GWDG  4.0K May 27 15:08 foo

In this case, if the -a option had not been used, the 78 MiB file .baz would have been completely overlooked. Notice that ls -lah also shows the total size of all files directly in the directory (not in subdirectories), the permissions, and the targets of symlinks.

One of the limitations of ls is that it does now show the total amount of space and inodes are used inside directories. For that, you need to use du. To get the block usage for one or more directories, run

du -sch DIR1 ... DIRN

and for inodes

du --inodes -sch DIR1 ... DIRN

where the -s option generates a summary of the total usage for each directory, the -c prints the sum-total of all the given directories, and -h converts the usage to human readable usage (e.g. converting 1073741824 bytes into 1G).

A quick check of the size of the current directory using this command would be

du -d 1 -h

where the option -d 1 limits the calculation to this directory.

An example with a user’s two directories in the scratch-grete data store would be

glogin9:~ $ du -sch /scratch-grete/usr/$USER /scratch-grete/tmp/$USER
79M     /scratch-grete/usr/gzadmfnord
60K     /scratch-grete/tmp/gzadmfnord
79M     total

and for inodes, it would be

glogin9:~ $ du --inodes -sch /scratch-grete/usr/$USER /scratch-grete/tmp/$USER
5       /scratch-grete/usr/gzadmfnord
1       /scratch-grete/tmp/gzadmfnord
6       total

Reducing Filesystem Usage

After determining where your high usage is, there are several things that can be done to reduce usage.

Clean up or delete no longer needed data

First, if the files/directories are no longer needed or are trivially regenerated, they can simply be deleted. In particular, the ~/.cache directory is a common culprit for exceeding the quota of one’s HOME directory since many tools cache files there and never clean up (conda seems to be the most common culprit). It is often safe to outright delete the ~/.cache directory or any of its sub-directories. Temporary files also tend to accrue in /scratch-*/tmp/USERNAME and /mnt/lustre-*/tmp/USERNAME directories, so they are good candidates for cleanup.

Move files to another Data Store

Second, the files/directories could be moved to somewhere else. Are you using your home directory for data that should really be in one of the project’s data stores? You have way more quota in your project directory or on cold storage (see below).

Another example would be moving files/directories needing less IO performance from an SSD based data store to a larger HDD based one.

Do you have files/directories that you want to keep, but have not used for a long time and probably won’t be needing again any time soon? Bundle/compress them (see below) and move them to cold storage (ceph-hdd) or tape. You could also download the data to your own machine/s to be re-uploaded later (obviously, special considerations must be taken for very large data sets).

Use Workspaces

Workspaces usually have much higher quota than permanent data stores. We offer workspaces on ceph-hdd-ws with very large quotas and extended durations in case you quickly need a lot more storage space for a limited amount of time.

Compress / Bundle your Data

Third, the files/directories could be compressed or bundled together in some way. Compression can reduce the block usage depending on the kind of data and the compression algorithm chosen, and can even improve read performance afterwards in some cases. Bundling many files/directories together can greatly reduce inode usage (e.g. packing a directory with a million files into a single TAR or ZIP file). Common methods include:

• tar, zip, etc. for bundling and compressing files and directories.
• Changing file formats to different ones that store the same data but smaller. For example, PNG will almost always store an image in less space than BMP will (most BMP images are either uncompressed or use a very weak compression algorithm).
• Concatenating files and making an index of the byte offsets and sizes of each to reduce the inode usage. A common example is concatenating many JPEG files making a MJPG (Motion JPEG) file which is supported by many tools directly (e.g. ffmpeg can read them just like any other video format).
• Putting the data from several files and/or directories in an HDF5 or NetCDF4 file with compression on the Datasets/Variables.
• Converting directories of thousands of images into a multi-page image format (e.g. multi-page TIFF) or a video. Note that videos can use the similarities between successive frames to compress. Be careful to choose an appropriate video codec for your data if you do this. You may want either a lossless codec (even if your original image frames are lossy), a codec put into its lossless mode, or a codec tuned to be low loss in order to avoid compounding losses (lossy JPEG then put through a lossy image codec). Encoding and decoding speed can be important as well.

Investigate File System usage via TUI frontend

On the login nodes, call ncdu with the root directory you want to analyze as the first argument:

ncdu <PATH>

There you can graphically see the space used up by each directory.

Bindings can be seen by pressing ?

The most important bindings are

To analyze inode usage, use the manual method described below.