Score-P

Score-P (Scalable Performance Measurement Infrastructure for Parallel Codes) is a scalable measurement framework for HPC applications typically written in C, C++ and Fortran.

The tool supports applications that employ both serial and a variety of parallel programming techniques. The parallel programming techniques currently supported are: Process-level parallelism (MPI, SHMEM), thread-level parallelism (OpenMP, Pthreads), accelerator-based parallelism (HIP, CUDA, OpenCL, OpenACC, OpenMP offloading), and possible combinations of these.

Score-P generates outputs in CUBE4 format for profiling data and OTF2 format for the tracing data which can also be analyzed with other performance tools such as Scalasca, Vampir and TAU, thereby providing a common measurement infrastructure. Score-P also supports event recording for popular memory management & I/O interfaces.

Using Score-P

To list the necessary dependant modules to be loaded and the available versions of Score-P on the cluster, use the command module spider scorep and follow the instructions from the output. For example, in our latest software stack, to load the module scorep/10.0, you first need to load the modules gcc/15.3.0 and openmpi/5.0.10.

To exemplify a typical workflow using Score-P, consider an MPI executable my_app generated from a source file my_src.c via the command:

mpicc -O2 my_src.c -o my_app

To run the application, for instance on 2 MPI ranks, you would usually use the command:

mpirun -n 2 [options] ./my_app

To instrument the above application using Score-P in an automatic way (recommended), prefix the compile command with the instrumenter scorep:

scorep mpicc -O2 my_src.c -o my_instrumented_app

This will insert special measurement calls into the application code at specific important points called ’events'.

To then run the instrumented application:

mpirun -n 2 [options] ./my_instrumented_app

Once the execution has finished, the tool creates a directory with the format scorep-YYYYMMDD_HHMM_XXXXXXXX with all the configured measurement data (YYYYMMDD and HHMM are the date in year-month-day format and time, respectively, when the measurement run was started, whereas XXXXXXXX is an additional identification number). The name of the directory can be modified by using the environment variable SCOREP_EXPERIMENT_DIRECTORY. By default, only profiling is enabled, i.e the output consists of the profile.cubex file. This file contains the profiling data as a performance report and can be viewed using the CUBE4 Performance Report Explorer, which is also available as a module.

module load cube
cube profile.cubex

To enable tracing, set the environment variable SCOREP_ENABLE_TRACING to 1. This will generate the necessary trace files among which the traces.otf2 file can be used to visualize the generated traces using a tool like Vampir. Please note that enabling tracing will introduce a significant measurement overhead.

To already get an idea of the potential size of traces and required memory, Score-P provides a mechanism called Scoring via the tool scorep-score. This tool takes the profile.cubex file as an input and is mainly used to define filters to control which parts of the application need to be traced and those to be excluded (e.g short frequent calls). This can be easily judged from the data provided by the tool as seen below.

To invoke the scoring tool, you may simply do:

scorep-score profile.cubex

Using a simple C code parallelized by MPI, shown below is an example output of the basic scoring mechanism:

Estimated aggregate size of event trace:                   38kB
Estimated requirements for largest trace buffer (max_buf): 2468 bytes
Estimated memory requirements (SCOREP_TOTAL_MEMORY):       4097kB
(hint: When tracing set SCOREP_TOTAL_MEMORY=4097kB to avoid intermediate flushes
 or reduce requirements using USR regions filters.)

flt     type max_buf[B] visits time[s] time[%] time/visit[us]  region
         ALL      2,467    764   19.94   100.0       26104.45  ALL
         MPI      2,138    556   19.66    98.6       35352.92  MPI
         COM        240    160    0.21     1.0        1297.49  COM
         USR         48     32    0.06     0.3        1817.74  USR
      SCOREP         41     16    0.02     0.1        1363.50  SCOREP

For further details about scoring and filtering, we recommend following the Score-P User guide

Using Score-P for Python applications

Considering the ubiquitous nature of Python in ML/AI applications, having a standard profiling and tracing interface becomes necessary to ensure optimum performance of the corresponding applications, especially when executing at scale.

Unfortunately, standard Python performance analysis tools have limitations when it comes to highly parallel programs. However, profiling and tracing parallel python codes is still possible through the Score-P Python bindings. Instrumentation is made possible by exploiting features of CPython and the official Score-P C bindings. The bindings have to be installed additionally as a Python module.

To profile a python file script.py, simply run:

python -m scorep <script.py>

For further details about the usage of these bindings, instructions to profile and/or trace your parallel python code, please visit their Github page.