CPU Partitions

Nodes in these partitons provide many CPU cores for parallelizing calculations.

Partitions

The NHR partitions following the naming scheme sizeCORES[suffix] where size indicates the amount of RAM (medium, standard, large, or huge), CORES indicates the number of cores, and suffix is only included to differentiate partitions with the same size and CORES. SCC partitions do not follow this scheme. The partitions are listed in the table below without hardware details.

Cluster	Partition	OS	Shared	Max. walltime	Max. nodes per job	Core-hours per node
NHR	medium96s	Rocky 8		12 hr	256	96
	medium96s:test	Rocky 8		1 hr	64	96
	standard96	Rocky 8		12 hr	256	96
	standard96:shared	Rocky 8	yes	48 hr	64	96
	standard96:test	Rocky 8		1 hr	64	96
	standard96s	Rocky 8		12 hr	256	96
	standard96s:shared	Rocky 8	yes	48 hr	1	96
	standard96s:test	Rocky 8		1 hr	64	96
	large96	Rocky 8		12 hr	2	192
	large96:shared	Rocky 8	yes	48 hr	1	192
	large96:test	Rocky 8		1 hr	2	192
	large96s	Rocky 8		12 hr	2	192
	large96s:shared	Rocky 8	yes	48 hr	1	192
	large96s:test	Rocky 8		1 hr	2	192
	huge96	Rocky 8		24 hr	256	192
	huge96s	Rocky 8		24 hr	1	192
	huge96s:shared	Rocky 8	yes	24 hr	1	192
	jupyter:cpu (jupyter)	Rocky 8	yes	24 hr	1	192
SCC	medium	Rocky 8	yes	48 hr	inf
	fat	Rocky 8	yes	48 hr	inf
	fat+	Rocky 8	yes	48 hr	inf
	int (jupyter)	Rocky 8	yes	48 hr	inf
	sgiz (reserved)	Rocky 8	yes	48 hr	inf
	fg (reserved)	Rocky 8	yes	30 days	inf
	soeding (reserved)	Rocky 8	yes	14 days	inf
	cidbn (reserved)	Rocky 8	yes	14 days	inf

Warning

The partitions you are allowed to use may be restricted by your kind of account and/or POSIX group. For example, the partitions marked as reserved in the table above are reserved-for/restricted-to specific research groups.

Info

JupyterHub sessions run on the partitions marked with jupyter in the table above.

The NHR nodes are grouped into subclusters of different hardware. The partitions for each NHR subcluster are listed in the table below where the parts of partition names in square brackets are optional (e.g. medium96s[:*] includes medium96s and medium96s:test).

NHR Subcluster	CPUs	Partitions
Emmy Phase 2	Intel Cascade Lake	`jupyter:cpu`, `standard96[:]`, `large96[:]`, `huge96`
Emmy Phase 3	Intel Sapphire Rapids	`medium96s[:]`, `standard96s[:]`, `large96s[:]`, `huge96s[:]`

The hardware for the different nodes in each partition are listed in the table below. Note that some partitions are heterogeneous, having nodes with different hardware. Additionally, many nodes are in more than one partition.

Partition	Nodes	CPU	RAM per node	Cores	SSD
medium96s	380	2 × Sapphire Rapids 8468	256 000 MB	96	yes
medium96s:test	164	2 × Sapphire Rapids 8468	256 000 MB	96	yes
standard96	853	2 × Cascadelake 9242	364 000 MB	96
	148	2 × Cascadelake 9242	364 000 MB	96	yes
standard96:shared	853	2 × Cascadelake 9242	364 000 MB	96
	138	2 × Cascadelake 9242	364 000 MB	96	yes
standard96:test	856	2 × Cascadelake 9242	364 000 MB	96
	140	2 × Cascadelake 9242	364 000 MB	96	yes
standard96s	220	2 × Sapphire Rapids 8468	514 000 MB	96	yes
standard96s:shared	220	2 × Sapphire Rapids 8468	514 000 MB	96	yes
standard96s:test	224	2 × Sapphire Rapids 8468	514 000 MB	96	yes
large96	12	2 × Cascadelake 9242	747 000 MB	96	yes
large96:shared	9	2 × Cascadelake 9242	747 000 MB	96	yes
large96:test	4	2 × Cascadelake 9242	747 000 MB	96	yes
large96s	13	2 × Sapphire Rapids 8468	1 030 000 MB	96	yes
large96s:shared	9	2 × Sapphire Rapids 8468	1 030 000 MB	96	yes
large96s:test	4	2 × Sapphire Rapids 8468	1 030 000 MB	96	yes
huge96	2	2 × Cascadelake 9242	1 522 000 MB	96	yes
huge96s	2	2 × Sapphire Rapids 8468	2 062 000 MB	96	yes
huge96:shared	2	2 × Sapphire Rapids 8468	2 062 000 MB	96	yes
jupyter:cpu	8	2 × Cascadelake 9242	364 000 MB	96
medium	94	2 × Cascadelake 9242	364 000 MB	96	yes
	12	2 × Broadwell E5-2650v4	512 000 MB	48	yes
	3	4 × Haswell E5-4620v3	1 500 000 MB	24	yes
	1	4 × Haswell E7-4809v3	2 048 000 MB	32	yes
fat	12	2 × Broadwell E5-2650v4	512 000 MB	48	yes
	4	4 × Haswell E5-4620v3	1 500 000 MB	24	yes
	1	4 × Haswell E7-4809v3	2 048 000 MB	32	yes
fat+	4	4 × Haswell E5-4620v3	1 500 000 MB	24	yes
	1	4 × Haswell E7-4809v3	2 048 000 MB	32	yes
int	2	2 × Cascadelake 9242	364 000 MB	96	yes
	11	2 × Skylake 6130	95 000 MB	32	yes
sgiz	11	2 × Skylake 6130	95 000 MB	32	yes
fg	8	2 × Zen3 EPYC 4713	512 000 MB	48	yes
soeding	7	2 × Zen2 EPYC 7742	1 000 000 MB	128	yes
cidbn	30	2 × Zen3 EPYC 7763	496 000 MB	128	yes

The CPUs

For partitions that have heterogeneous hardware, you can give Slurm options to request the particular hardware you want. For CPUs, you can specify the kind of CPU you want by passing a -C/--constraint option to slurm to get the CPUs you want. Use -C ssd or --constraint=ssd to request a node with a local SSD on the NHR cluster, and -C local or --constraint local to request a node with a local SSD on the SCC cluster. See Slurm for more information.

The CPUs, the options to request them, and some of their properties are give in the table below.

CPU	Cores	`-C` option	Architecture
AMD Zen3 EPYC 7763	64	`milan` or `zen3`	`zen3`
AMD Zen3 EPYC 4713	24	`milan` or `zen3`	`zen3`
AMD Zen2 EPYC 7742	64	`rome` or `zen2`	`zen2`
Intel Sapphire Rapids Xeon Platinum 8468	48	`sapphirerapids`	`sapphirerapids`
Intel Cascadelake Xeon Platinum 9242	48	`cascadelake`	`cascadelake`
Intel Skylake Xeon Gold 6130	16	`skylake`	`skylake_avx512`
Intel Broadwell Xeon E5-2650 V4	12	`broadwell`	`broadwell`
Intel Haswell Xeon E5-4620 V3	10	`haswell`	`haswell`
Intel Haswell Xeon E7-4809 V3	8	`haswell`	`haswell`

The entire CPU table (downloadable)

CPU Partitions

Cluster	Partition	Cores per node	Core-hours per node	Max. walltime	Max. nodes per job	Nodes	RAM per node	OS	CPU	-C option	Cores per CPU	SSD	Shared
NHR	medium96s	96	96.0	12 hr	256.0	380	256 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	no
NHR	medium96s:test	96	96.0	1 hr	64.0	164	256 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	no
NHR	standard96	96	96.0	12 hr	256.0	853	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	no	no
NHR	standard96	96	96.0	12 hr	256.0	148	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	no
NHR	standard96:shared	96	96.0	48 hr	64.0	853	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	no	yes
NHR	standard96:shared	96	96.0	48 hr	64.0	138	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	no
NHR	standard96:test	96	96.0	1 hr	64.0	856	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	no	no
NHR	standard96:test	96	96.0	1 hr	64.0	140	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	no
NHR	standard96s	96	96.0	12 hr	256.0	220	514 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	no
NHR	standard96s:shared	96	96.0	48 hr	1.0	220	514 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	yes
NHR	standard96s:test	96	96.0	1 hr	64.0	224	514 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	NaN
NHR	large96	96	192.0	12 hr	2.0	12	747 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	no
NHR	large96:shared	96	192.0	48 hr	1.0	9	747 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	yes
NHR	large96:test	96	192.0	1 hr	2.0	4	747 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	no
NHR	large96s	96	192.0	12 hr	2.0	13	1030 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	no
NHR	large96s:shared	96	192.0	48 hr	1.0	9	1030 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	yes
NHR	large96s:test	96	192.0	1 hr	2.0	4	1030 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	no
NHR	huge96	96	192.0	24 hr	256.0	2	1522 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	no
NHR	huge96s	96	192.0	24 hr	1.0	2	2062 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	no
NHR	huge96s:shared	96	192.0	24 hr	1.0	2	2062 GB	Rocky 8	2 × Sapphire Rapids 8468	sapphirerapids	48	yes	yes
NHR	jupyter:cpu\n(jupyter)	96	192.0	24 hr	1.0	8	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	NaN	yes
SCC	medium	96	NaN	48 hr	inf	94	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	yes
SCC	medium	48	NaN	48 hr	inf	12	512 GB	Rocky 8	2 × Broadwell E5-2650v4	broadwell	12	yes	no
SCC	medium	24	NaN	48 hr	inf	3	1500 GB	Rocky 8	4 × Haswell E5-4620v3	haswell	10	yes	no
SCC	medium	32	NaN	48 hr	inf	1	2048 GB	Rocky 8	4 × Haswell E7-4809v3	haswell	8	yes	no
SCC	fat	48	NaN	48 hr	inf	12	512 GB	Rocky 8	2 × Broadwell E5-2650v4	broadwell	12	yes	yes
SCC	fat	24	NaN	48 hr	inf	4	1500 GB	Rocky 8	4 × Haswell E5-4620v3	haswell	10	yes	no
SCC	fat	32	NaN	48 hr	inf	1	2048 GB	Rocky 8	4 × Haswell E7-4809v3	haswell	8	yes	no
SCC	fat+	24	NaN	48 hr	inf	4	1500 GB	Rocky 8	4 × Haswell E5-4620v3	haswell	10	yes	yes
SCC	fat+	32	NaN	48 hr	inf	1	2048 GB	Rocky 8	4 × Haswell E7-4809v3	haswell	8	yes	no
SCC	int (jupyter)	96	NaN	48 hr	inf	2	364 GB	Rocky 8	2 × Cascadelake 9242	cascadelake	48	yes	yes
SCC	int (jupyter)	32	NaN	48 hr	inf	11	95 GB	Rocky 8	2 × Skylake 6130	skylake	16	yes	no
SCC	sgiz (reserved)	32	NaN	48 hr	inf	11	95 GB	Rocky 8	2 × Skylake 6130	skylake	16	yes	yes
SCC	fg (reserved)	48	NaN	30 days	inf	8	512 GB	Rocky 8	2 × Zen3 EPYC 4713	milan or zen3	24	yes	yes
SCC	soeding (reserved)	128	NaN	14 days	inf	7	1000 GB	Rocky 8	2 × Zen2 EPYC 7742	rome or zen2	64	yes	yes
SCC	cidbn (reserved)	128	NaN	14 days	inf	30	496 GB	Rocky 8	2 × Zen3 EPYC 7763	milan or zen3	64	yes	yes

Hardware Totals

The total nodes, cores, and RAM for each cluster and sub-cluster are given in the table below.

Cluster	Sub-cluster	Nodes	Cores	RAM (TiB)
NHR	Emmy Phase 2	1,022	98,112	362.8
	Emmy Phase 3	411	39,456	173.4
	TOTAL	1,433	137,568	537.2
SCC	main	113	9,920	46.8
	CIDBN	30	3,840	14.1
	sa	8	384	3.9
	hh	7	896	6.6
	sgiz	11	352	1.0
	TOTAL	169	15,392	72.6