Job runtimes and QoS

Job Walltime

The maximum runtime is set per partition and can be seen either on the system with sinfo or here. There is no minimum walltime (we cannot stop your jobs from finishing, obviously), but a walltime of at least 1 hour is strongly recommended. Our system is optimized for high performance, not high throughput. A large amount of smaller, shorter jobs induces a lot of overhead as each job has a prolog, setting up the enviroment, an epilog for cleaning up and bookkeeping which can put a lot of load on the scheduler. The occasional short job is fine, but if you submit larger amounts of jobs that finish (or crash) quickly, we might have to intervene and temporarily suspend your account. If you have lots of smaller workloads, please consider combining them into a single job that runs for at least 1 hour. A tool often recommended to help with issues like this (among other useful features) is Jobber.

Jobs that need more than 12 hours of runtime

The default runtime limit on most partitions is 12 hours. On systems where your compute time is limited (you can check with the sbalance command), we will only refund jobs that run up to 12 hours, beyond that you are running the application at your own risk. But you can use the --time option in your sbatch script to request up to 48 hours of runtime.

Jobs that need more than 48 hours of runtime

Most compute partitions have a maximum wall time of 48 hours. Under exceptional circumstances, it is possible to get a time extension for individual jobs (past 48 hours) by writing a ticket during normal business hours. To apply for an extension, please write a support request, containing the Job ID, the username, the project and the reason why the extension is necessary. Alternatively - under even more exceptional circumstances and also via mail request, including username, project ID and reason - permanent access to Slurm Quality-Of-Service (QoS) levels can be granted, which permit a longer runtime for jobs (up to 96 hours) but have additional restrictions regarding job size (e.g. number of nodes).

Please make sure you have checked out all of the following suggestions before you request access to the 96h qos:

1. Can you increase the parallelization of your application to reduce the runtime?
2. Can you use checkpointing to periodically write out the intermediate state of your calculation and restart from that in a subsequent job?
3. Are you running the code on the fastest available nodes?
4. Is it possible to use GPU acceleration to speed up your calculation?
5. Are you using the latest software revision (in contrast to e.g. using hlrn-tmod or rev/xx.yy)?

Dependent & Restartable Jobs - How to pass the wall time limit

If your simulation is restartable, it might be handy to automatically trigger a follow-up job. Simply provide the ID of the previous job as an additional sbatch argument:

# submit a first job, extract job id
jobid=$(sbatch --parsable job1.sbatch)
 
# submit a second job with dependency: starts only if the previous job terminates successfully)
jobid=$(sbatch --parsable --dependency=afterok:$jobid job2.sbatch)
 
# submit a third job with dependency: starts only if the previous job terminates successfully)
jobid=$(sbatch --parsable --dependency=afterok:$jobid job3.sbatch)