Linking with the netcdf library
Problem
After linking a program with a netcdf library some users report error messages like this:
./nctest: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory
One my find this error message by searching the web. Hence, one is in good company. Here we present methods to use the tools available at HLRN to produce running binaries. First of all an important statement:
Neither the netcdf nor the hdf5 module have any effect on your binary during runtime. It is useless to load them before running a program.
The following example program can be used for testing only. Don’t ask for a sophisticated science background. The example uses FORTRAN-90 notation.
download the example
To compile the code, the netcdf data- and function types must be included by a use statement. The compiler needs to know, where the fortran-modules are located. Instead of digging into installation details, we use the result of the script nf-config, which is part of the netcdf-suite. To make it available, load a netcdf module:
module load gcc/8.3.0
module load netcdf/gcc.8/4.7.3
Note, compiler version and netcdf version must fit, otherwise the fortran netcdf module cannot be included. We can check this with
nf-config --fc
fortran .mod files from different major gfortran versions are incompatible.
Dynamic linking with the fortran netcdf library
Serial programms
Now you can compile
gfortran -c -I`nf-config --includedir` test.f90
Instead of digging even more into the installation details, we use the result of nf-config in the link step:
gfortran -o nctest *.o `nf-config --flibs` -lm
To understand the details, just issue the commands nf-config –includedir and nf-config –flibs separately.
Now you may unload both, the compiler and the netcdf module as well. The binary runs anyway, since the path to the netcdf- and hdf5- libraries is stored in the binary. This information was extracted from the environment variable LD_RUN_PATH that is set when loading a netcdf module. Note, the path to the hdf5 libraries linked to the netcdf library is inherited by our binary. The effect of LD_RUN_PATH is broken, if the linker option rpath is used for other purposes. Say, your example program links another library located in /mybla. Checkout
gfortran -o nctest -Wl,-rpath=/mybla *.o `nf-config --flibs` -lm
ldd nctest
The netcdf-libraries as well as the compiler specific libraries are not found any more. Here we notice, the compiler module also uses LD_RUN_PATH to set the path to the compiler specific libraries. So handle LD_RUN_PATH with care not to disturb this important feature! The correct way would be:
gfortran -o nctest -Wl,-rpath="/mybla:$LD_RUN_PATH" *.o `nf-config --flibs` -lm
Hint, you may use readelf to see more details on the internal binary structure. Watch out for the library sections.
MPI programms
Now we need to use the compiler wrappers to include mpi.h and link the mpi libraries automatically. Here, rpath is used internally, which breaks the automatic propagation of the information of the paths, where the netcdf library resides. This concerns also the location of compiler specific libraries like libgcc_s.so.1 or libquadmath.so.0 . From that there may arise problems, since libquadmath.so.0 coming with gcc.9 contains more symbols than the system version. (This allows usage of the netcdf library build with gcc.8 together with gcc.9 .) Hence, we have to use the rpath option to provide the information on the library path and propagate it to the linker. Load the module files:
module load gcc/9.2.0
module load netcdf/gcc.8/4.7.3
module load openmpi/gcc.9/3.1.5
and check LD_RUN_PATH:
echo $LD_RUN_PATH
/sw/comm/openmpi/3.1.5/skl/gcc/lib:/sw/dataformats/netcdf/gcc.8.3.0/4.7.3/skl/lib:/sw/compiler/gcc/9.2.0/skl/lib64/
Compile:
mpifort -c -I`nf-config --includedir` test.f90
mpifort -o nctest *.o `nf-config --flibs` -Wl,-rpath=$LD_RUN_PATH -lm
To check the true content of the binary, unload the modules first
module unload openmpi/gcc.9/3.1.5
module unload gcc/9.2.0
module unload netcdf/gcc.8/4.7.3
ldd nctest
With readelf -a more details can be explored:
readelf -a nctest | less
….
Dynamic section at offset 0x2d58 contains 37 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libnetcdff.so.7]
0x0000000000000001 (NEEDED) Shared library: [libnetcdf.so.15]
0x0000000000000001 (NEEDED) Shared library: [libgfortran.so.5]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [libmpi_usempif08.so.40]
0x0000000000000001 (NEEDED) Shared library: [libmpi_usempi_ignore_tkr.so.40]
0x0000000000000001 (NEEDED) Shared library: [libmpi_mpifh.so.40]
0x0000000000000001 (NEEDED) Shared library: [libmpi.so.40]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libquadmath.so.0]
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000001d (RUNPATH) Library runpath: [/sw/comm/openmpi/3.1.5/skl/gcc/lib:/sw/dataformats/netcdf/gcc.8.3.0/4.7.3/skl/lib:/sw/compiler/gcc/9.2.0/skl/lib64/:/sw/tools/hwloc/1.1.13/skl/lib]
...
Static linking with the fortran netcdf library
There is no simple way to find all required libraries, but information on the libraries can be gathered from the file libnetcdff.la made by libtool for dynamic linking.
NETCDF_LIB=`nc-config --libdir`
cat $NETCDF_LIB/libnetcdff.la | grep dependency_libs
We configure the paths to the relevan libraries by hand:
NETCDF_LIB=`nc-config --libdir`
HDF5_LIB=/sw/dataformats/hdf5/1.8.20/skl/gcc.8.2.0.hlrn/lib
SZIP_LIB=/sw/dataformats/szip/2.1/skl/gcc.8.2.0.hlrn/lib
LIBS="$NETCDF_LIB/libnetcdff.a $NETCDF_LIB/libnetcdf.a $HDF5_LIB/libhdf5_hl.a $HDF5_LIB/libhdf5.a $SZIP_LIB/libsz.a -lpthread -lcurl -lz -lm -ldl"
Now compile and link
gfortran -fPIC -c -I`nf-config --includedir` -Bstatic test.f90
gfortran -o nctest *.o $LIBS
and have fun.