Python

Python is a powerful programming and scripting language. Its standard library already covers a wide range of tools and facilities. It can be extended by many third-party software packages created for different purposes.

This page is organized into 4 sections, where the simplest use cases are documented at the top and the most complex ones at the bottom:

1. Python Versions and Module Files
2. Python Virtual Environments
3. The uv Python Package and Project Manager
4. Installing conda Packages with miniforge3

If you are new to Python in general, we recommend to start reading at the top and not skip over any sections.

Python Versions and Module Files

The software installed in the current revision 25.04 uses the python/3.11.9 module. If you want to use python packages provided by these modules or extend them, you should use this module to provide your python interpreter.

Standard Python (No Additional Packages)

To load the default python interpreter and start executing python scripts, please run the following commands:

module load gcc
module load python

It is then as simple as python ./script.py to execute your first python script on the system.

Alternatively you can also make your python script executable (chmod +x script.py) and add the following she-bang line as the first line of your script:

#!/usr/bin/env python

Then you can execute it directly as ./script.py after loading the python interpreter. If you are inside a virtual environment (documented below), this shortcut will also work.

Python Virtual Environments

The python ecosystem has a large problem with complex dependencies and version incompatibilities. To ensure that updating the dependencies of one software package does not break a different one, you should isolate them in so-called virtual environments. The simplest method is to use the built-in venv feature of python.

Where to Store Your Environments

In the past we have recommended using the $WORK filesystem for NHR users to store their virtual environments for lack of better options. This is not a good idea since the underlying parallel filesystem is not designed to store many small files and perform well with the large amount of metadata this creates. Using a large number of inodes on this filesystem is discouraged.

Please check the following table to find a suitable location:

Creating a Virtual Environment

You can create a new virtual environment called foo in the current directory:

module load gcc
module load python/3.11.9
python -m venv foo

Now you can activate it by executing source ./foo/bin/activate. This will also add a (foo) prefix to your shell prompt.

You can install some extra packages from the Python Package Index (PyPI):

(foo) $ pip install lmfit matplotlib seaborn

Using a Virtual Environment

To run some python application that uses these modules, you can just activate the environment and then execute the script with the python command:

source ./foo/bin/activate
python myscript.py

When you are done manipulating an environment, you can run deactivate to leave it.

The uv Python Package and Project Manager

Please read and understand the above section Python Virtual Environments first! It contains valuable hints about the recommended usage of virtual environments on our system.

If you don’t want to interact with the software installed in the module system but just install your own python packages and also pick your own version of the python interpreter, you can use uv.

Creating a Python Project with uv

Here is an example that turns the current directory into a new python project:

module load uv
uv python list # Lists all available python versions. In our case the list included the version 3.12.11.
uv python install 3.12.11
uv python pin 3.12.11
uv init
uv add lmfit matplotlib seaborn

To run a script in this project, just switch to the project directory and run:

module load uv
uv run script.py

Please check out the official documentation for more information.

Using uv as a Replacement for venv/pip

You can also directly use uv as a replacement for venv and pip:

module load uv
uv venv
uv pip install lmfit matplotlib seaborn
uv run script.py

Please check out the official documentation for more information about the pip interface.

Running Python Tools with uvx

Many python application are used as tools. For this the uvx command can be used.

In this example we reformat a python script using black:

uvx black script.sh

Please check out the official documentation for more information about using tools.

Installing conda Packages with miniforge3

Please read and understand the above section Python Virtual Environments first! It contains valuable hints about the recommended usage of virtual environments on our system.

Note that the miniforge3 module has replaced miniconda and anaconda on our system, since it only uses the free conda-forge channel by default. Installing software with miniconda or anaconda (from the default channel) would require purchasing an expensive commercial license (this applies to academic research and commercial usage, class-room usage remains free).

Loading miniforge3 and Preparing an Environment

In order to use Python and Conda environments, load the module miniforge3 and create an environment.

This example loads the module, creates a new environment called myenv in the current directory, and activates it using the source command:

module load miniforge3
conda create --prefix ./myenv python=3.12
source activate ./myenv

Once this is done, you are able to use Python normally, as you would on a personal computer with commands such as:

conda install -y numpy scipy matplotlib
pip install pillow

source $CONDASH

Loading the Environment in a Batch Script

To load an environment in your batch script you repeat the above steps for activation:

This includes the source command and the path:

module load miniforge3
source activate ./myenv
# OR
module load miniforge3
source $CONDASH
conda activate ./myenv

Make sure you are in the right directory, or better yet, use the absolute path.

realpath ./myenv