Accounting Core-h

The NHR centers follow the NHR-wide regulations and account the resource usage in the unit core hours (“coreh” or sometimes “core-hr” for short). Running compute jobs using the batch system will result in the accumulation of used core hours. Projects have a quarterly limit of how many core hours can be used until jobs are no longer being started.

Every node in each of the available compute partitions has a specific charge rate. The current rates can be found in the table for the CPU partitions and also for the GPU partitions.

Job Charge

Each node in our system has an associated cost in core hours that a job consumes per hour when allocating this node during its runtime. We call this a charge rate. Both our CPU and GPU partitions come in two different variants when it comes to how this value is calculated.

In the detailed overview of available GPU and CPU partitions, you will see the attribute “Shared”.

How the charge rate is calculated depends on this value.

Charge rate for non-shared partitions

For most partitions, nodes are exclusive, meaning they can only be used by a single user at a time. Using an exclusive node consumes the resources of the entire node and is accounted as a full node. This is the case even when fewer resources (e.g. less than the full number of CPU cores) were requested, and even though in such cases only the requested amount of resources are actually usable.

In other words: On a partition that does not have “Shared”: “yes”, a job will always be assigned entire nodes and charged accordingly. This is independent of the actual number of CPUs/cores/tasks requested for each node. Calculating the charge rate in this case needs to consider all resources that are available in the node:

• CPU partitions: The charge rate is calculated from the number of cores in a node, multiplied by a “Core-hr per Core” factor that you can find here.
• GPU partitions: The charge rate is calculated from the number of GPUs in a node, multiplied by the “Core-hr per GPU” factor that you can find here.

Charge rate for shared partitions

A shared node can be used by multiple users and therefore, only the allocated number of cores are accounted. When using a GPU node, only the allocated GPU(s) are taken into account, meaning allocating and using a GPU node will only cost the core hours for the GPU(s), not the CPUs. Note that allocated GPUs count for the entire runtime of your job, not just the time you actually spent running calculations on them.

In other words: On “Shared”: “yes” partitions, compute jobs can allocate parts of a node. What your job costs directly depends on the number of allocated CPU cores or GPUs, depending on the partition type:

• CPU partitions: The charge rate is calculated from the number of allocated cores for each node, multiplied by a “Core-hr per Core” factor that you can find here.
• GPU partitions: The charge rate is calculated from the number of GPUs allocated per node, multiplied by a “Core-hr per GPU” factor that you can find here.

Cost of entire jobs

The scheduler, Slurm, keeps track of how many core hours each compute job consumed. The charge for an entire batch job on our systems is the number of core hours consumed and is calculated from the number of nodes reserved for the job, the wallclock time used by the job, and the charge rate for each of the job’s nodes.

For a batch job with

• num nodes,
• running with a wallclock time of t hours, and
• on a partition with a charge rate charge_p

the job charge charge_j yields

charge_j = num * t * charge_p

Examples:

• Assume a job running on 2 nodes in partition medium96s for 12 hours. Nodes in this partition have 96 cores and the “Core-hr per Core” factor is 0.75. The charge rate becomes charge_p = 96 * 0.75 = 72. The total charge for the entire job is then charge_j = 2 * 12 * 72 = 1728.
• Assume a job running on grete:shared on 1 node, allocating 2 GPUs for 10 hours. Since this is a shared partition, only the allocated GPUs are billed. The “Core-hours per GPU” factor for that partition is 150, hence the charge rate per node becomes charge_p = 2 * 150 = 300. The total charge for the entire job is then charge_j = 1 * 10 * 300 = 3000.
• Assume the same GPU job as above, but accidentally submitted to grete. Since this is a non shared partition, it does not matter that only two GPUs where requested. The charge rate for the node is calculated as charge_p = 4 * 150 = 600 and for the entire job charge_j = 1 * 10 * 600 = 6000.

You can use the sacct command to view information about runtime and allocated compute resources for past jobs.

JobID             ElapsedRaw                            AllocTRES
------------ --------------- ------------------------------------
12345678               43230  billing=384,cpu=384,mem=400G,node=2
12345678.ba+           43231              cpu=384,mem=400G,node=2
12345678.ex+           43237  billing=384,cpu=384,mem=400G,node=2
12345678.0             43233              cpu=384,mem=400G,node=2

Checking the balance

Every project and every user has an account in Slurm’s database, which stores the already used core hours and upper limits they are allowed to use. To get an overview of your consumption, we provide a tool called sbalance. It prints your current balance and provides additional useful information.

sbalance: Prints the current balance of users and accounts as well as their hierarchy.

 -u, --user          string  Specify the user to query
 -a, --account       string  Specify the account to query
 -c, --withchildren  bool    Whether to display the child accounts / users
 -l, --limit         bool    Whether to display the limit instead of the used credits
 -r, --remaining     bool    Whether to display the remaining instead of the used credits
 -s, --short         bool    No tree, no progressbar, no limits or units. Only print the used credit amount.
 -n, --no-hints      bool    Disable printing hints.

The output can look like this:

You are part of project nim12345:

└─ projects                                      (2.00 / unlimited*) Mcore-hr
  └─ extern                                      (1.84 / unlimited*) Mcore-hr
    └─ nhr                                       (1.42 / unlimited*) Mcore-hr
      └─ nhr_ni                                  (430.55 / unlimited*) kcore-hr
        └─ nim12345 [#############            ]  (0.79 / 1.62) Mcore-hr
          └─ u12345                              (0.00 / unlimited*) core-hr

*) Unlimited projects and accounts are still affected
by limits imposed on the accounts in the tree above them.

Accounting Types

In Göttingen, we no longer have independent personal accounts, only project-specific users that are part of a project in the HPC Project Portal. Most projects have an allocation of coreh for the entire project and each user submitting a job to the cluster will consume the resources of the project they are associated to. The only exception here are NHR Testaccounts, which are part of a “Testaccounts” project for the respective federal state of the user’s home institution, but are accounted for individually.

The main project types relevant in terms of accounting are NHR projects and KISSKI projects, both have different ways of allocating resources, as detailed below.

NHR Projects

A compute project is granted with a quarterly contingent of core hours. At the beginning of each quarter, the count of used core hours is reset to zero and the limit is set to the granted number of core hours, according to the funding decision for the given compute project. A project account’s limit is at least 4 x 300.000 core hours per year.

Unused core hours are transferred to the subsequent quarter, for a single time. Coreh unused in Q1 will be transferred to Q2, but at the end of Q2 only the coreh from Q2 can be transferred to Q3. Should the project still have coreh from Q1 left at the end of Q2, they will not be transferred again.

To give a concrete example, project nim12345 has been granted 400 kcoreh for each quarter of the year. They used 200 kcoreh in Q1, so their remaining 200k will be transferred to Q2, meaning the limit for Q2 will be 600 kcoreh instead of 400. In Q2, they had some difficulties and only used 50k, so the remaining balance at the end of Q2 was 550 kcoreh. Since 150 of them were already transferred from Q1, only the 400k for Q2 are transferred to Q3, resulting in a limit of 800 kcoreh.

In case of problems with the accounting for your project, please contact the NHR Support. This might include the

• application for additional core hours
• movement of core hours between quarters
• correction of wrong limits set for your account

KISSKI projects

KISSKI projects are accounted in GPU hours, but on the HPC system, sbalance will still show you your allocated compute time in coreh. Without going into detail, the default limit for a project is 25.000 GPU hours, which is about 3.750.000 core-hours. The maximum that can be requested when applying for a KISSKI project is 50.000 GPU hours. Using the table in GPU partitions, it is possible to calculate the associated core-hours for your project. Simply look for the “Core-hours per GPU” entry for the kisski partition. An example for such a calculation can be found here.

Terminology

The unit used is the number of hours and cores used. It can be expressed as:

• core hours
• coreh
• core-hr
• kcoreh = kilo core hours = 1000 core hours
• Mcoreh = mega core hours = 1000 000 core hours

Using the SI abbreviations is useful as a shorthand and might be used in printouts, examples, support answers, etc.