Commit 83bece39 authored by Jacob Finkenrath's avatar Jacob Finkenrath

Merge branch 'r2.2-dev' of https://repository.prace-ri.eu/git/UEABS/ueabs into r2.2-dev-qcd

parents 8e847bd0 b2808717
......@@ -37,8 +37,8 @@ There is currently no active support for non-CUDA accelerator platforms.
For the UEABS benchmark version 2.2, the following versions of GPAW were tested:
* CPU-based:
* Version 1.5.3 as this one is the last of the 1.5 branch and since the GPU version
is derived from 1.5.2.
* Version 1.5.2 as this one is the last of the 1.5 branch and since the GPU version
is derived from this version.
* Version 20.1.0, the most recent version during the development of the UEABS
2.2 benchmark suite.
* GPU-based: There is no official release or version number. The UEABS 2.2 benchmark
......@@ -60,10 +60,18 @@ Input file: [benchmark/1_S_carbon-nanotube/input.py](benchmark/1_S_carbon-nanotu
### Case M: Copper filament
A ground state calculation for a copper filament in vacuum. By default uses a
2x2x3 FCC lattice with 71 atoms (freely adjustable) and ScaLAPACK for
parallelisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
3x4x4 FCC lattice with 71 atoms (freely adjustable through the variables `x`,
`y` and `z` in the input file) and ScaLAPACK for
parallellisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
Input file: [benchmark/2_M_carbon-nanotube/input.py](benchmark/2_M_copper-filament/input.py)
Input file: [benchmark/2_M_copper-filament/input.py](benchmark/2_M_copper-filament/input.py)
The benchmark was tested using 1000 and 1024 cores. For some core configurations, one may
get error messages similar to ``gpaw.grid_descriptor.BadGridError: Grid ... to small
for ... cores``. If one really wants to run the benchmark for those number of cores,
one needs to adapt the values of `x`, `y` and `z` in `input.py`. However, this
changes the benchmark so results cannot be compared easily with benchmark runs for
different values of these variables.
### Case L: Silicon cluster
......@@ -72,7 +80,7 @@ cluster has a radius of 15Å (freely adjustable) and consists of 702 atoms,
and ScaLAPACK is used for parallelisation. Expected to scale up to 1000 nodes
and/or 10000 MPI tasks.
Input file: [benchmark/3_L_carbon-nanotube/input.py](benchmark/3_L_silicon-cluster/input.py)
Input file: [benchmark/3_L_silicon-cluster/input.py](benchmark/3_L_silicon-cluster/input.py)
## Mechanics of building the benchmark
......@@ -82,18 +90,37 @@ last version with the old numbering. In 2019 the development team switched
to a version numbering scheme based on year, month and patchlevel, e.g.,
19.8.1 for the second version released in August 2019.
A further major change affecting both the build process and the mechanics of running
the benchmark happened in version 20.1.0. Versions up to and including 19.8.1 use a
wrapper executable `gpaw-python` that replaces the Python interpreter (it internally
links to the libpython library) and provides the MPI functionality. From version 20.1.0
the standard Python interpreter is used and the MPI functionality is included in the `_gpaw.so`
shared library, though there is still an option in the build process (not tested for
the UEABS benchmarks) to generate that wrapper instead.
Another change is in the Python packages used to install GPAW. Versions up to
and including 19.8.1 use the `distutils` package while versions 20.1.0 and later
are based on `setuptools`. This does affect the installation process.
GPAW for a while supports two different ways to run in parallel distributed memory mode:
* Using a wrapper executable `gpaw-python` that replaces the Python interpreter (it internally
links to the libpython library) and that provides the MPI functionality.
* Using the standard Python interpreter, including the MPI functionality in the
`_gpaw.so` shared library.
In the `distutils`-based versions, the wrapper script approach is the default behaviour,
while in the `setuptools`-based versions, the approach using the standard Python interpreter
is the preferred one in the manual. Even though the code in the `setuptools`-based
versions still includes the option to use the wrapper script approach, it does not
work in the tested version 20.1.0.
### Available instructions
The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) only contains the
[installation instructions](https://wiki.fysik.dtu.dk/gpaw/index.html) for the current version.
For the installation instructions with a list of dependencies for older versions,
download the code (see below) and look for the file `doc/install.rst` or go to the
[GPAW GitLab](https://gitlab.com/gpaw), select the tag for the desired version and
view the file `doc/install.rst`.
The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) also provides some
[platform specific examples](https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html).
### List of dependencies
GPAW is Python code (3.5 or newer) but it also contains some C code for some performance-critical
GPAW is Python code but it also contains some C code for some performance-critical
parts and to interface to a number of libraries on which it depends.
Hence GPAW has the following requirements:
......@@ -101,16 +128,20 @@ Hence GPAW has the following requirements:
* BLAS, LAPACK, BLACS and ScaLAPACK. ScaLAPACK is optional for GPAW, but mandatory
for the UEABS benchmarks. It is used by the medium and large cases and optional
for the small case.
* Python 3.5 or newer
* Python. GPAW 1.5.2 requires
Python 2.7 or 3.4-3.7, GPAW 19.8.1 requires 3.4-3.7, GPAW 20.1.0 Python 3.5-3.8
and GPAW 20.10.0 Python 3.6-3.9.
* Mandatory Python packages:
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 19.8.1/20.1.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 19.8.1/20.1.0)
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [FFTW](http://www.fftw.org) is highly recommended. As long as the optional libvdwxc
component is not used, the MKL FFTW wrappers can also be used. Recent versions of
GPAW can even show good performance using just the NumPy-provided FFT routines provided
GPAW also show good performance using just the NumPy-provided FFT routines provided
that NumPy has been built with a highly optimized FFT library.
* [LibXC](https://www.tddft.org/programs/libxc/) 3.X or 4.X. LibXC is a library
of exchange-correlation functions for density-functional theory
* [LibXC](https://www.tddft.org/programs/libxc/) 2.X or newer for GPAW 1.5.2,
3.X or 4.X for GPAW 19.8.1, 20.1.0 and 20.10.0. LibXC is a library
of exchange-correlation functions for density-functional theory. None of the
versions currently mentions LibXC 5.X as officially supported.
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW
* Check the release notes of GPAW as the releases of ASE and GPAW should match.
......@@ -128,9 +159,17 @@ Hence GPAW has the following requirements:
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
In addition, the GPU version needs:
* CUDA toolkit
* NVIDIA CUDA toolkit
* [PyCUDA](https://pypi.org/project/pycuda/)
Installing GPAW also requires a number of standard build tools on the system, including
* [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
configure script for libxc
* [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
the configure process of libxc produces very misleading
error messages that do not immediately point to libtool missing.
* [GNU make](https://www.gnu.org/software/make/)
### Download of GPAW
......@@ -155,21 +194,16 @@ git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
### Install
Official generic [installation instructions](https://wiki.fysik.dtu.dk/gpaw/install.html)
and
[platform specific examples](https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html)
are provided in the [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/).
Crucial for the configuration of GPAW is a proper `customize.py` (GPAW 19.8.1 and
earlier) or `siteconfig.py` (GPAW 20.1.0 and later) file. The defaults used by GPAW
may not offer optimal performance and the automatic detection of the libraries also
fails on some systems.
The UEABS repository contains additional instructions:
* [general instructions](installation.md) - Under development
* [general instructions](build/build-cpu.md)
* [GPGPUs](build/build-cuda.md) - To check
Example [build scripts](build/examples/) are also available for some PRACE
Example [build scripts](build/examples/) are also available for some PRACE and non-PRACE
systems.
......@@ -187,7 +221,9 @@ right from this repository.
### Running the benchmarks
#### Versions up to and including 19.8.1 of GPAW
#### Using the `gpaw-python` wrapper script
This is the default approach for versions up to and including 19.8.1 of GPAW
These versions of GPAW come with their own wrapper executable, `gpaw-python`,
to start a MPI-based GPAW run.
......@@ -199,7 +235,9 @@ properly with the resource manager. E.g., on Slurm systems, use
srun gpaw-python input.py
```
#### GPAW 20.1.0 (and likely later)
#### Using the regular Python interpreter and parallel GPAW shared library
This is the default method for GPAW 20.1.0 (and likely later).
The wrapper executable `gpaw-python` is no longer available in the default parallel
build of GPAW. There are now two different ways to start GPAW.
......@@ -229,8 +267,8 @@ would do.
Example [job scripts](scripts/) (`scripts/job-*.sh`) are provided for
different PRACE systems that may offer a helpful starting point.
TODO: Update the examples.
*TODO: Update the examples as testing on other systems goes on.*
## Verification of Results
......@@ -240,9 +278,9 @@ TODO.
### Case M: Copper filament
TODO.
TODO. Convergence problems.
### Case L: Silicon cluster
TODO.
TODO. Get the medium case to run before spending time on the large one.
This diff is collapsed.
Patch written by Kurt Lust, UAntwerp for PRACE IP UEABS
- Backports a change to c/xc/tpss.c from 19.8.1 to 1.5.2.
- Backports a change to c/xc/revtpss.c from 19.8.1 to 1.5.2.
--- gpaw-1.5.2/c/xc/tpss.c.orig 2020-10-22 16:15:29.000000000 +0200
+++ gpaw-1.5.2/c/xc/tpss.c 2020-10-22 16:37:41.000000000 +0200
@@ -427,12 +427,11 @@
int nder = (nspin==XC_UNPOLARIZED) ? 1 : 3;
for(i=0; i<nder; i++){
if(nspin==XC_UNPOLARIZED) dauxdsigma[i] /= 2.;
- double dCdsigma[i];
- dCdsigma[i]= dCdcsi*dcsidsigma[i];
+ double dCdsigma = dCdcsi*dcsidsigma[i];
/* partial derivatives*/
- de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma[i] * zsq
- - zsq * (dens * dCdsigma[i] * aux + (1.0 + C) * dauxdsigma[i]);
+ de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma * zsq
+ - zsq * (dens * dCdsigma * aux + (1.0 + C) * dauxdsigma[i]);
}
}
--- gpaw-1.5.2/c/xc/revtpss.c.orig 2020-10-22 16:15:29.000000000 +0200
+++ gpaw-1.5.2/c/xc/revtpss.c 2020-10-22 17:12:40.000000000 +0200
@@ -231,12 +231,11 @@
int nder = (p->common.nspin==XC_UNPOLARIZED) ? 1 : 3;
for(i=0; i<nder; i++){
if(p->common.nspin==XC_UNPOLARIZED) dauxdsigma[i] /= 2.;
- double dCdsigma[i];
- dCdsigma[i]= dCdcsi*dcsidsigma[i];
-
+ double dCdsigma = dCdcsi*dcsidsigma[i];
+
/* partial derivatives*/
- de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma[i] * zsq
- - zsq * (dens * dCdsigma[i] * aux + (1.0 + C) * dauxdsigma[i]);
+ de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma * zsq
+ - zsq * (dens * dCdsigma * aux + (1.0 + C) * dauxdsigma[i]);
}
}
# NEMO
## Summary Version
1.0
1.1
## Purpose of Benchmark
......@@ -35,90 +36,92 @@ The model is implemented in Fortran 90, with pre-processing (C-pre-processor). I
```
./make_xios --arch local
```
Files for the PRACE Tier-0 systems are available under [architecture_files](architecture_files) folder.
Note that XIOS requires `Netcdf4`. Please load the appropriate `HDF5` and `NetCDF4` modules. You might have to change the path in the configuration file.
Note that XIOS requires `Netcdf4`. Please load the appropriate `HDF5` and `NetCDF4` modules. If path to these models are not loaded, you might have to change the path in the configuration file.
### Building NEMO
1. Download the XIOS source code:
```
svn co https://forge.ipsl.jussieu.fr/nemo/svn/NEMO/releases/release-4.0
```
2. Copy and setup the appropriate architecture file in the arch folder. The following changes are recommended:
2. Copy and setup the appropriate architecture file in the arch folder. Files for the PRACE Tier-0 systems are available under [architecture_files](architecture_files) folder. The following changes are recommended for the GNU compilers:
```
a. add the `-lnetcdff` and `-lstdc++` flags to NetCDF flags
b. using `mpif90` which is a MPI binding of `gfortran-4.9`
c. add `-cpp` and `-ffree-line-length-none` to Fortran flags
d. swap out `gmake` with `make`
```
3. Then build the executable with the following command
```
./makenemo -m MY_CONFIG -r GYRE_XIOS -n MY_GYRE add_key "key_nosignedzero"
```
4. Apply the patch as described here to measure step time :
3. Apply the patch as described here to measure step time :
```
https://software.intel.com/en-us/articles/building-and-running-nemo-on-xeon-processors
```
You may also use [nemogcm.F90](nemogcm.F90) by replacing it with `src/OCE/nemogcm.F90`
4. go to `cfgs` folder and add `GYRE_testing OCE TOP` line to `refs_cfg.txt` file.
Then:
```
mkdir GYRE_testing
rsync -arv GYRE_PISCES/* GYRE_testing/
mv GYRE_testing/cpp_GYRE_PISCES.fcm GYRE_testing/cpp_GYRE_testing.fcm
In GYRE_testing/cpp_GYRE_testing.fcm file replace key_top with key_nosignedzero
```
5. Build the executable with the following command
```
../makenemo -m MY_CONFIG -r GYRE_testing
```
## Mechanics of Running Benchmark
### Prepare input files
cd MY_GYRE/EXP00
cd GYRE_testing/EXP00
sed -i '/using_server/s/false/true/' iodef.xml
sed -i '/&nameos/a ln_useCT = .false.' namelist_cfg
sed -i '/&namctl/a nn_bench = 1' namelist_cfg
sed -i '/ln_bench/s/false/true/' namelist_cfg
### Run the experiment interactively
mpirun -n 4 ../BLD/bin/nemo.exe -n 2 $PATH_TO_XIOS/bin/xios_server.exe
mpirun -n 4 nemo : -n 2 $PATH_TO_XIOS/bin/xios_server.exe
### GYRE configuration with higher resolution
Modify configuration (for example for the test case A):
```
rm -f time.step solver.stat output.namelist.dyn ocean.output slurm-* GYRE_* mesh_mask_00*
jp_cfg=4
rm -f time.step solver.stat output.namelist.dyn ocean.output slurm-* GYRE_*
sed -i -r \
-e 's/^( *nn_itend *=).*/\1 21600/' \
-e 's/^( *nn_stock *=).*/\1 21600/' \
-e 's/^( *nn_write *=).*/\1 1000/' \
-e 's/^( *jp_cfg *=).*/\1 '"$jp_cfg"'/' \
-e 's/^( *jpidta *=).*/\1 '"$(( 30 * jp_cfg +2))"'/' \
-e 's/^( *jpjdta *=).*/\1 '"$(( 20 * jp_cfg +2))"'/' \
-e 's/^( *jpiglo *=).*/\1 '"$(( 30 * jp_cfg +2))"'/' \
-e 's/^( *jpjglo *=).*/\1 '"$(( 20 * jp_cfg +2))"'/' \
-e 's/^( *nn_itend *=).*/\1 101/' \
-e 's/^( *nn_write *=).*/\1 4320/' \
-e 's/^( *nn_GYRE *=).*/\1 48/' \
-e 's/^( *rn_rdt *=).*/\1 1200/' \
namelist_cfg
```
## Verification of Results
The GYRE configuration is set through the `namelist_cfg` file. The horizontal resolution is determined by setting `jp_cfg` as follows:
The GYRE configuration is set through the `namelist_cfg` file. The horizontal resolution is determined by setting `nn_GYRE` as follows:
```
Jpiglo = 30 × jp_cfg + 2
Jpjglo = 20 × jp_cfg + 2
Jpiglo = 30 × nn_GYRE + 2
Jpjglo = 20 × nn_GYRE + 2
```
In this configuration, we use a default value of 30 ocean levels, depicted by `jpk=31`. The GYRE configuration is an ideal case for benchmark tests as it is very simple to increase the resolution and perform both weak and strong scalability experiment using the same input files. We use two configurations as follows:
In this configuration, we use a default value of 30 ocean levels, depicted by `jpkglo=31`. The GYRE configuration is an ideal case for benchmark tests as it is very simple to increase the resolution and perform both weak and strong scalability experiment using the same input files. We use two configurations as follows:
Test Case A:
```
jp_cfg = 128 suitable up to 1000 cores
Number of Days: 20
Number of Time steps: 1440
nn_GYRE = 48 suitable up to 1000 cores
Number of Time steps: 101
Time step size: 20 mins
Number of seconds per time step: 1200
```
Test Case B:
```
jp_cfg = 256 suitable up to 20,000 cores.
Number of Days (real): 80
Number of time step: 4320
nn_GYRE = 192 suitable up to 20,000 cores.
Number of time step: 101
Time step size(real): 20 mins
Number of seconds per time step: 1200
```
We performed scalability test on 512 cores and 1024 cores for test case A. We performed scalability test for 4096 cores, 8192 cores and 16384 cores for test case B.
Both these test cases can give us quite good understanding of node performance and interconnect behavior.
We switch off the generation of mesh files by setting the `flag nn_mesh = 0` in the `namelist_ref` file. Also `using_server = false` is defined in `io_server` file.
<!--We switch off the generation of mesh files by setting the `flag nn_mesh = 0` in the `namelist_ref` file.
Also `using_server = false` is defined in `io_server` file.-->
We report the performance in step time which is the total computational time averaged over the number of time steps for different test cases.
This helps us to compare systems in a standard manner across all combinations of system architectures.
......
# generic ifort compiler options for JUWELS
#
# NCDF_HOME root directory containing lib and include subdirectories for netcdf4
# HDF5_HOME root directory containing lib and include subdirectories for HDF5
# XIOS_HOME root directory containing lib for XIOS
# OASIS_HOME root directory containing lib for OASIS
#
# NCDF_INC netcdf4 include file
# NCDF_LIB netcdf4 library
# XIOS_INC xios include file (taken into accound only if key_iomput is activated)
# XIOS_LIB xios library (taken into accound only if key_iomput is activated)
# OASIS_INC oasis include file (taken into accound only if key_oasis3 is activated)
# OASIS_LIB oasis library (taken into accound only if key_oasis3 is activated)
#
# FC Fortran compiler command
# FCFLAGS Fortran compiler flags
# FFLAGS Fortran 77 compiler flags
# LD linker
# LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries
# FPPFLAGS pre-processing flags
# AR assembler
# ARFLAGS assembler flags
# MK make
# USER_INC complete list of include files
# USER_LIB complete list of libraries to pass to the linker
# CC C compiler used to compile conv for AGRIF
# CFLAGS compiler flags used with CC
#
# Note that:
# - unix variables "$..." are accpeted and will be evaluated before calling fcm.
# - fcm variables are starting with a % (and not a $)
#
%NCDF_HOME /gpfs/software/juwels/stages/2019a/software/netCDF/4.6.3-ipsmpi-2019a.1/
%NCDF_HOME2 /gpfs/software/juwels/stages/2019a/software/netCDF-Fortran/4.4.5-ipsmpi-2019a.1/
%HDF5_HOME /gpfs/software/juwels/stages/2019a/software/HDF5/1.10.5-ipsmpi-2019a.1/
%CURL /gpfs/software/juwels/stages/2019a/software/cURL/7.64.1-GCCcore-8.3.0/lib/
%XIOS_HOME /p/project/prpb86/nemo2/xios-2.5/
%OASIS_HOME /not/defined
%HDF5_LIB -L%HDF5_HOME/lib -L%CURL -lhdf5_hl -lhdf5
%GCCLIB
%NCDF_INC -I%NCDF_HOME/include -I%NCDF_HOME2/include -I%HDF5_HOME/include
%NCDF_LIB -L%NCDF_HOME/lib %HDF5_LIB -L%CURL -L%NCDF_HOME2/lib -lnetcdff -lnetcdf -L%GCCLIB -lstdc++ -lz -lcurl -lgpfs
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios -L%GCCLIB -lstdc++
%OASIS_INC -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
%OASIS_LIB -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip
%CPP icc -E
%FC mpifort
%FCFLAGS -O3 -r8 -funroll-all-loops -traceback
%FFLAGS %FCFLAGS
%LD mpifort
%LDFLAGS -lstdc++ -lifcore -O3 -traceback
%FPPFLAGS -P -C -traditional
%AR ar
%ARFLAGS -r
%MK make
%USER_INC %XIOS_INC %OASIS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %OASIS_LIB %NCDF_LIB
%CC cc
%CFLAGS -O0
# generic gfortran compiler options for linux M100
#
# NCDF_HOME root directory containing lib and include subdirectories for netcdf4
# HDF5_HOME root directory containing lib and include subdirectories for HDF5
# XIOS_HOME root directory containing lib for XIOS
# OASIS_HOME root directory containing lib for OASIS
#
# NCDF_INC netcdf4 include file
# NCDF_LIB netcdf4 library
# XIOS_INC xios include file (taken into accound only if key_iomput is activated)
# XIOS_LIB xios library (taken into accound only if key_iomput is activated)
# OASIS_INC oasis include file (taken into accound only if key_oasis3 is activated)
# OASIS_LIB oasis library (taken into accound only if key_oasis3 is activated)
#
# FC Fortran compiler command
# FCFLAGS Fortran compiler flags
# FFLAGS Fortran 77 compiler flags
# LD linker
# LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries
# FPPFLAGS pre-processing flags
# AR assembler
# ARFLAGS assembler flags
# MK make
# USER_INC complete list of include files
# USER_LIB complete list of libraries to pass to the linker
# CC C compiler used to compile conv for AGRIF
# CFLAGS compiler flags used with CC
#
# Note that:
# - unix variables "$..." are accpeted and will be evaluated before calling fcm.
# - fcm variables are starting with a % (and not a $)
#
%NCDF_HOME /cineca/prod/opt/libraries/netcdf/4.7.3/gnu--8.4.0/
%NCDF_HOME2 /cineca/prod/opt/libraries/netcdff/4.5.2/gnu--8.4.0/
%HDF5_HOME /cineca/prod/opt/libraries/hdf5/1.12.0/gnu--8.4.0/
%XIOS_HOME /m100_work/Ppp4x_5387/xios-2.5/
%OASIS_HOME /not/defined
%HDF5_LIB -L%HDF5_HOME/lib -lhdf5_hl -lhdf5
%GCCLIB /cineca/prod/opt/compilers/gnu/8.4.0/none/lib64/
%NCDF_INC -I%NCDF_HOME/include -I%NCDF_HOME2/include -I%HDF5_HOME/include
%NCDF_LIB -L%NCDF_HOME/lib %HDF5_LIB -L%NCDF_HOME2/lib -lnetcdff -lnetcdf -L%GCCLIB -lstdc++ -lz -lcurl -lgpfs
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios -L%GCCLIB -lstdc++
%OASIS_INC -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
%OASIS_LIB -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip
%CPP cpp -Dkey_nosignedzero
%FC mpif90
%FCFLAGS -fdefault-real-8 -fno-second-underscore -O3 -funroll-all-loops -fcray-pointer -cpp -ffree-line-length-none -Dgfortran
%FFLAGS %FCFLAGS
%LD %FC
%LDFLAGS
%FPPFLAGS -P -C -traditional -x f77-cpp-input
%AR ar
%ARFLAGS rs
%MK make
%USER_INC %XIOS_INC %OASIS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %OASIS_LIB %NCDF_LIB
%CC cc
%CFLAGS -O0
# generic ifort compiler options for SuperMUC
#
# NCDF_HOME root directory containing lib and include subdirectories for netcdf4
# HDF5_HOME root directory containing lib and include subdirectories for HDF5
# XIOS_HOME root directory containing lib for XIOS
# OASIS_HOME root directory containing lib for OASIS
#
# NCDF_INC netcdf4 include file
# NCDF_LIB netcdf4 library
# XIOS_INC xios include file (taken into accound only if key_iomput is activated)
# XIOS_LIB xios library (taken into accound only if key_iomput is activated)
# OASIS_INC oasis include file (taken into accound only if key_oasis3 is activated)
# OASIS_LIB oasis library (taken into accound only if key_oasis3 is activated)
#
# FC Fortran compiler command
# FCFLAGS Fortran compiler flags
# FFLAGS Fortran 77 compiler flags
# LD linker
# LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries
# FPPFLAGS pre-processing flags
# AR assembler
# ARFLAGS assembler flags
# MK make
# USER_INC complete list of include files
# USER_LIB complete list of libraries to pass to the linker
# CC C compiler used to compile conv for AGRIF
# CFLAGS compiler flags used with CC
#
# Note that:
# - unix variables "$..." are accpeted and will be evaluated before calling fcm.
# - fcm variables are starting with a % (and not a $)
#
%NCDF_HOME /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/netcdf/4.6.1-intel-rdopmwr/
%NCDF_HOME2 /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/netcdf-fortran/4.4.4-intel-mq54rwz/
%HDF5_HOME /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/hdf5/1.10.2-intel-726msh6/
%CURL /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/curl/7.60.0-gcc-u7vewcb/lib/
%XIOS_HOME /hppfs/work/pn68so/di67wat/NEMO/xios-2.5/
%OASIS_HOME /not/defined
%HDF5_LIB -L%HDF5_HOME/lib -L%CURL -lhdf5_hl -lhdf5
%GCCLIB
%NCDF_INC -I%NCDF_HOME/include -I%NCDF_HOME2/include -I%HDF5_HOME/include
%NCDF_LIB -L%NCDF_HOME/lib %HDF5_LIB -L%CURL -L%NCDF_HOME2/lib -lnetcdff -lnetcdf -L%GCCLIB -lstdc++ -lz -lcurl -lgpfs
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios -L%GCCLIB -lstdc++
%OASIS_INC -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
%OASIS_LIB -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip
%CPP icc -E
%FC mpiifort
%FCFLAGS -O3 -r8 -funroll-all-loops -traceback
%FFLAGS %FCFLAGS
%LD mpiifort
%LDFLAGS -lstdc++ -lifcore -O3 -traceback
%FPPFLAGS -P -C -traditional
%AR ar
%ARFLAGS -r
%MK make
%USER_INC %XIOS_INC %OASIS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %OASIS_LIB %NCDF_LIB
%CC cc
%CFLAGS -O0
module load GCC/8.3.0
module load PGI/19.10-GCC-8.3.0
module load Intel/2019.5.281-GCC-8.3.0
module load ParaStationMPI/5.4
module load HDF5/1.10.5
module load netCDF/4.6.3
module load netCDF-Fortran/4.4.5
module load cURL
module load Perl
################################################################################
################### Projet XIOS ###################
################################################################################
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90 -nofor-main
%BASE_CFLAGS -ansi -w
%PROD_CFLAGS -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -O2
%DEBUG_CFLAGS -g
%BASE_FFLAGS -D__NONE__ -ffree-line-length-none
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2
%DEBUG_FFLAGS -g
%BASE_INC -D__NONE__
%BASE_LD -lstdc++
%CPP cpp
%FPP cpp -P
%MAKE make
NETCDF_INCDIR="-I$NETCDF_INC_DIR -I$NETCDFF_INC_DIR"
NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L$NETCDF_LIB_DIR -L$NETCDFF_LIB_DIR"
NETCDF_LIB="-lnetcdff -lnetcdf"
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_INCDIR="-I $HDF5_INC_DIR"
HDF5_LIBDIR="-L $HDF5_LIB_DIR"
HDF5_LIB="-lhdf5_hl -lhdf5 -lz -lcurl"
BOOST_INCDIR="-I $BOOST_INC_DIR"
BOOST_LIBDIR="-L $BOOST_LIB_DIR"
BOOST_LIB=""
OASIS_INCDIR="-I$PWD/../../oasis3-mct/BLD/build/lib/psmile.MPI1"
OASIS_LIBDIR="-L$PWD/../../oasis3-mct/BLD/lib"
OASIS_LIB="-lpsmile.MPI1 -lscrip -lmct -lmpeu"
module load gnu
module load szip
module load zlib
module load spectrum_mpi
module load hdf5
module load netcdf
module load netcdff
################################################################################
################### Projet XIOS ###################
################################################################################
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90
%BASE_CFLAGS -ansi -w
%PROD_CFLAGS -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -O2 -traceback
%DEBUG_CFLAGS -DBZ_DEBUG -g -traceback -fno-inline
%BASE_FFLAGS -D__NONE__ -ffree-line-length-none
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2 -traceback
%DEBUG_FFLAGS -g -traceback
%BASE_INC -D __NONE__
%BASE_LD -lstdc++
%CPP cpp
%FPP cpp -P
%MAKE make
NETCDF_INCDIR="-I$NETCDF_INC_DIR -I$NETCDFF_INC_DIR"
NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L$NETCDF_LIB_DIR -L$NETCDFF_LIB_DIR"
NETCDF_LIB="-lnetcdff -lnetcdf"
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_INCDIR="-I $HDF5_INC_DIR"
HDF5_LIBDIR="-L $HDF5_LIB_DIR"
HDF5_LIB="-lhdf5_hl -lhdf5 -lz -lcurl"
BOOST_INCDIR="-I $BOOST_INC_DIR"
BOOST_LIBDIR="-L $BOOST_LIB_DIR"
BOOST_LIB=""
OASIS_INCDIR="-I$PWD/../../oasis3-mct/BLD/build/lib/psmile.MPI1"
OASIS_LIBDIR="-L$PWD/../../oasis3-mct/BLD/lib"
OASIS_LIB="-lpsmile.MPI1 -lscrip -lmct -lmpeu"
module load slurm_setup
module load hdf5
module load netcdf
module load netcdf-fortran
################################################################################
################### Projet XIOS ###################
################################################################################
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90 -nofor-main
%BASE_CFLAGS -ansi -w
%PROD_CFLAGS -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -O2
%DEBUG_CFLAGS -g
%BASE_FFLAGS -D__NONE__ -ffree-line-length-none
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2
%DEBUG_FFLAGS -g
%BASE_INC -D__NONE__
%BASE_LD -lstdc++
%CPP cpp
%FPP cpp -P
%MAKE make
NETCDF_INCDIR="-I$NETCDF_INC_DIR -I$NETCDFF_INC_DIR"
NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L $NETCDF_LIB_DIR -L $NETCDFF_LIB_DIR"