gpaw/README.md

@@ -37,8 +37,8 @@ There is currently no active support for non-CUDA accelerator platforms.
 
 For the UEABS benchmark version 2.2, the following versions of GPAW were tested:
   * CPU-based:
-      * Version 1.5.3 as this one is the last of the 1.5 branch and since the GPU version 
-        is derived from 1.5.2.
+      * Version 1.5.2 as this one is the last of the 1.5 branch and since the GPU version 
+        is derived from this version.
       * Version 20.1.0, the most recent version during the development of the UEABS 
         2.2 benchmark suite.
   * GPU-based: There is no official release or version number. The UEABS 2.2 benchmark 
@@ -60,10 +60,18 @@ Input file: [benchmark/1_S_carbon-nanotube/input.py](benchmark/1_S_carbon-nanotu
 ### Case M: Copper filament
 
 A ground state calculation for a copper filament in vacuum. By default uses a
-2x2x3 FCC lattice with 71 atoms (freely adjustable) and ScaLAPACK for
-parallelisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
+3x4x4 FCC lattice with 71 atoms (freely adjustable through the variables `x`,
+`y` and `z` in the input file) and ScaLAPACK for
+parallellisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
 
-Input file: [benchmark/2_M_carbon-nanotube/input.py](benchmark/2_M_copper-filament/input.py)
+Input file: [benchmark/2_M_copper-filament/input.py](benchmark/2_M_copper-filament/input.py)
+
+The benchmark was tested using 1000 and 1024 cores. For some core configurations, one may 
+get error messages similar to ``gpaw.grid_descriptor.BadGridError: Grid ... to small 
+for ... cores``. If one really wants to run the benchmark for those number of cores, 
+one needs to adapt the values of `x`, `y` and `z` in `input.py`. However, this 
+changes the benchmark so results cannot be compared easily with benchmark runs for 
+different values of these variables.
 
 ### Case L: Silicon cluster
 
@@ -72,7 +80,7 @@ cluster has a radius of 15Å (freely adjustable) and consists of 702 atoms,
 and ScaLAPACK is used for parallelisation. Expected to scale up to 1000 nodes
 and/or 10000 MPI tasks.
 
-Input file: [benchmark/3_L_carbon-nanotube/input.py](benchmark/3_L_silicon-cluster/input.py)
+Input file: [benchmark/3_L_silicon-cluster/input.py](benchmark/3_L_silicon-cluster/input.py)
 
 
 ## Mechanics of building the benchmark
@@ -82,18 +90,37 @@ last version with the old numbering. In 2019 the development team switched
 to a version numbering scheme based on year, month and patchlevel, e.g.,
 19.8.1 for the second version released in August 2019.
 
-A further major change affecting both the build process and the mechanics of running 
-the benchmark happened in version 20.1.0. Versions up to and including 19.8.1 use a 
-wrapper executable `gpaw-python` that replaces the Python interpreter (it internally 
-links to the libpython library) and provides the MPI functionality. From version 20.1.0 
-the standard Python interpreter is used and the MPI functionality is included in the `_gpaw.so` 
-shared library, though there is still an option in the build process (not tested for 
-the UEABS benchmarks) to generate that wrapper instead.
+Another change is in the Python packages used to install GPAW. Versions up to
+and including 19.8.1 use the `distutils` package while versions 20.1.0 and later
+are based on `setuptools`. This does affect the installation process.
+
+GPAW for a while supports two different ways to run in parallel distributed memory mode:
+  * Using a wrapper executable `gpaw-python` that replaces the Python interpreter (it internally 
+    links to the libpython library) and that provides the MPI functionality.
+  * Using the standard Python interpreter, including the MPI functionality in the 
+    `_gpaw.so` shared library.
+In the `distutils`-based versions, the wrapper script approach is the default behaviour,
+while in the `setuptools`-based versions, the approach using the standard Python interpreter
+is the preferred one in the manual. Even though the code in the `setuptools`-based 
+versions still includes the option to use the wrapper script approach, it does not 
+work in the tested version 20.1.0.
+
+### Available instructions
+
+The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) only contains the
+[installation instructions](https://wiki.fysik.dtu.dk/gpaw/index.html) for the current version.
+For the installation instructions with a list of dependencies for older versions, 
+download the code (see below) and look for the file `doc/install.rst` or go to the
+[GPAW GitLab](https://gitlab.com/gpaw), select the tag for the desired version and
+view the file `doc/install.rst`.
+
+The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) also provides some
+[platform specific examples](https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html).
 
 
 ### List of dependencies
 
-GPAW is Python code (3.5 or newer) but it also contains some C code for some performance-critical
+GPAW is Python code but it also contains some C code for some performance-critical
 parts and to interface to a number of libraries on which it depends.
 
 Hence GPAW has the following requirements:
@@ -101,16 +128,20 @@ Hence GPAW has the following requirements:
   * BLAS, LAPACK, BLACS and ScaLAPACK. ScaLAPACK is optional for GPAW, but mandatory 
     for the UEABS benchmarks. It is used by the medium and large cases and optional 
     for the small case.
-  * Python 3.5 or newer
+  * Python. GPAW 1.5.2 requires
+    Python 2.7 or 3.4-3.7, GPAW 19.8.1 requires 3.4-3.7, GPAW 20.1.0 Python 3.5-3.8
+    and GPAW 20.10.0 Python 3.6-3.9.
   * Mandatory Python packages:
-      * [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 19.8.1/20.1.0)
-      * [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 19.8.1/20.1.0)
+      * [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
+      * [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
   * [FFTW](http://www.fftw.org) is highly recommended. As long as the optional libvdwxc 
     component is not used, the MKL FFTW wrappers can also be used. Recent versions of 
-    GPAW can even show good performance using just the NumPy-provided FFT routines provided 
+    GPAW also show good performance using just the NumPy-provided FFT routines provided 
     that NumPy has been built with a highly optimized FFT library.
-  * [LibXC](https://www.tddft.org/programs/libxc/) 3.X or 4.X. LibXC is a library
-    of exchange-correlation functions for density-functional theory
+  * [LibXC](https://www.tddft.org/programs/libxc/) 2.X or newer for GPAW 1.5.2, 
+    3.X or 4.X for GPAW 19.8.1, 20.1.0 and 20.10.0. LibXC is a library
+    of exchange-correlation functions for density-functional theory. None of the
+    versions currently mentions LibXC 5.X as officially supported.
   * [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package 
     from the same group that develops GPAW
       * Check the release notes of GPAW as the releases of ASE and GPAW should match.
@@ -128,9 +159,17 @@ Hence GPAW has the following requirements:
         [LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html) 
 
 In addition, the GPU version needs:
-  * CUDA toolkit
+  * NVIDIA CUDA toolkit
   * [PyCUDA](https://pypi.org/project/pycuda/)
 
+Installing GPAW also requires a number of standard build tools on the system, including
+  * [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the 
+    configure script for libxc
+  * [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found, 
+    the configure process of libxc produces very misleading
+    error messages that do not immediately point to libtool missing.
+  * [GNU make](https://www.gnu.org/software/make/)
+
 
 ### Download of GPAW
 
@@ -155,21 +194,16 @@ git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
 
 ### Install
 
-Official generic [installation instructions](https://wiki.fysik.dtu.dk/gpaw/install.html)
-and
-[platform specific examples](https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html)
-are provided in the [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/).
-
 Crucial for the configuration of GPAW is a proper `customize.py` (GPAW 19.8.1 and 
 earlier) or `siteconfig.py` (GPAW 20.1.0 and later) file. The defaults used by GPAW 
 may not offer optimal performance and the automatic detection of the libraries also 
 fails on some systems.
 
 The UEABS repository contains additional instructions:
-  * [general instructions](installation.md) - Under development
+  * [general instructions](build/build-cpu.md)
   * [GPGPUs](build/build-cuda.md) - To check
 
-Example [build scripts](build/examples/) are also available for some PRACE
+Example [build scripts](build/examples/) are also available for some PRACE and non-PRACE
 systems.
 
 
@@ -187,7 +221,9 @@ right from this repository.
 
 ### Running the benchmarks
 
-#### Versions up to and including 19.8.1 of GPAW
+#### Using the `gpaw-python` wrapper script
+
+This is the default approach for versions up to and including 19.8.1 of GPAW
 
 These versions of GPAW come with their own wrapper executable, `gpaw-python`, 
 to start a MPI-based GPAW run.
@@ -199,7 +235,9 @@ properly with the resource manager. E.g., on Slurm systems, use
 srun gpaw-python input.py
 ```
 
-#### GPAW 20.1.0 (and likely later)
+#### Using the regular Python interpreter and parallel GPAW shared library
+
+This is the default method for GPAW 20.1.0 (and likely later).
 
 The wrapper executable `gpaw-python` is no longer available in the default parallel
 build of GPAW. There are now two different ways to start GPAW.
@@ -229,8 +267,8 @@ would do.
 
 Example [job scripts](scripts/) (`scripts/job-*.sh`) are provided for
 different PRACE systems that may offer a helpful starting point.
-TODO: Update the examples.
 
+*TODO: Update the examples as testing on other systems goes on.*
 
 ## Verification of Results
 
@@ -240,9 +278,9 @@ TODO.
 
 ### Case M: Copper filament
 
-TODO.
+TODO. Convergence problems.
 
 ### Case L: Silicon cluster
 
-TODO.
+TODO. Get the medium case to run before spending time on the large one.
 

gpaw/build/build-CPU.md

+# Detailed GPAW installation instructions on non-acclerated systems
+
+These instructions are in addition to the brief instructions in [README.md](../README.md).
+
+## Detailed dependency list
+
+### Libraries and Python interpreter
+
+GPAW needs (for the UEABS benchmarks)
+  * [Python](https://www.python.org/): GPAW 1.5.2 supports Python 2.7 and 3.4-3.7. 
+    GPAW 19.8.1 needs Python 3.4-3.7 and GPAW 20.1.0 requires Python 3.5-3.8.
+  * [MPI library](https://www.mpi-forum.org/)
+  * [LibXC](https://www.tddft.org/programs/libxc/). GPAW 1.5.2 requires LibXC 1.5.2
+    or later. GPAW 19.8.1 and 20.1.0 need LibXC 3.x or 4.x.
+  * (Optimized) [BLAS](http://www.netlib.org/blas/) and 
+    [LAPACK](http://www.netlib.org/lapack/) libraries. 
+    There are both commercial and free and open source versions of these libraries. 
+    Using the [reference implementation of BLAS from netlib](http://www.netlib.org/blas/) 
+    will give very poor performance. Most optimized LAPACK libraries actually only
+    optimize a few critical routines while the remaining routines are compiled from 
+    the reference version. Most processor vendors for HPC machines and system vendors
+    offer optmized versions of these libraries.
+  * [ScaLAPACK](http://www.netlib.org/scalapack/) and the underlying communication
+    layer [BLACS](http://www.netlib.org/blacs/).
+  * [FFTW](http://www.fftw.org/) or compatible FFT library. 
+    For the UEABS benchmarks, the double precision, non-MPI version is sufficient. 
+    GPAW also works with the
+    [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) 
+    FFT routines when using the FFTW wrappers provided with that product.
+
+For the GPU version, the following packages are needed in addition to the packages
+above:
+  * CUDA toolkit
+  * [PyCUDA](https://pypi.org/project/pycuda/)
+
+Optional components of GPAW that are not used by the UEABS benchmarks:
+  * [libvdwxc](https://gitlab.com/libvdwxc/libvdwxc), a portable C library 
+    of density functionals with van der Waals interactions for density functional theory.
+    This library does not work with the MKL FFTW wrappers as it needs the MPI version
+    of the FFTW libraries too.
+  * [ELPA](https://elpa.mpcdf.mpg.de/), 
+    which should improve performance for large systems when GPAW is used in
+    [LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html) 
+
+
+### Python packages
+
+GPAW needs 
+  * [wheel](https://pypi.org/project/wheel/) is needed in most (if not all) ways of 
+    installing the packages from source.
+  * [NumPy](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
+      * Installing NumPy from source will also require 
+        [Cython](https://pypi.org/project/Cython/)
+      * GPAW 1.5.2 is not fully compatible with NumPy 1.19.x. Warnings about the use
+        of deprecated constructs will be shown.
+  * [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
+  * [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package 
+    from the same group that develops GPAW. Required versions are 3.17.0 or later for
+    GPAW 1.5.2 and 3.18.0 or later for GPAW 19.8.1 or 20.1.0. 
+    ASE has a couple of dependendencies 
+    that are not needed for running the UEABS benchmarks. However, several Python
+    package install methods will trigger the installation of those packages, and 
+    with them may require a chain of system libraries.
+      * ASE does need NumPy and SciPy, but these are needed anyway for GPAW.
+      * [matplotlib](https://pypi.org/project/matplotlib/), at least version 2.0.0.
+        This package is optional and not really needed to run the benchmarks.
+        Matplotlib pulls in a lot of other dependencies. When installing ASE with pip,
+        it will try to pull in matplotlib and its dependencies
+          * [pillow](https://pypi.org/project/Pillow/) needs several exgternal 
+            libraries. During the development of the benchmarks, we needed at least
+            zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even 
+            though the pillow documentation claimed that libjpeg was optional, 
+            it refused to install without.
+          * [kiwisolver](https://pypi.org/project/kiwisolver/): Contains C++-code
+          * [pyparsing](https://pypi.org/project/pyparsing/)
+          * [Cycler](https://pypi.org/project/Cycler/), which requires
+              * [six](https://pypi.org/project/six/)
+          * [python-dateutil](https://pypi.org/project/python-dateutil/), which also 
+            requires
+              * [six](https://pypi.org/project/six/)
+      * [Flask](https://pypi.org/project/Flask/) is an optional dependency of ASE
+        that is not automatically pulled in by `pip` in versions of ASE tested during
+        the development of this version of the UEABS. It has a number of dependencies
+        too:
+          * [Jinja2](https://pypi.org/project/Jinja2/)
+              * [MarkupSafe](https://pypi.org/project/MarkupSafe/), contains some C 
+                code
+          * [itsdangerous](https://pypi.org/project/itsdangerous/)
+          * [Werkzeug](https://pypi.org/project/Werkzeug/)
+          * [click]()
+
+
+## Tested configurations
+
+  * Python
+  * Libraries used during the installation of Python:
+      * ncurses 6.2
+      * libreadline 8.0, as it makes life easy when using the command line
+        interface of Python (and in case of an EasyBuild Python, because EasyBuild
+        requires it)
+      * libffi 3.3
+      * zlib 1.2.11
+      * OpenSSL 1.1.1g, but only when EasyBuild was used and requires it.
+      * SQLite 3.33.0, as one of the tests in some versions of GPAW requires it to
+        succeed.
+      * Python will of course pick up several other libraries that it might find on
+        the system. The benchmark installation was tested on a system with very few
+        development packages of libraries installed in the system image. Tcl/Tk
+        and SQLite3 development packages in particular where not installed, so the
+        standard Python library packages sqlite3 and tkinter were not fully functional.
+  * Python packages
+      * wheel
+      * Cython
+      * NumPy
+      * SciPy
+      * ASE
+      * GPAW
+
+The table below give the combinations of major packages Python, NumPy, SciPy, ASE and 
+GPAW that were tested:
+
+| Python | NumPy  | SciPy | ASE    | GPAW    |
+|:-------|:-------|:------|:-------|:--------|
+| 3.7.9  | 1.18.5 | 1.4.1 | 3.17.0 | 1.5.2   |
+| 3.7.9  | 1.18.5 | 1.4.1 | 3.18.2 | 19.8.1  |
+| 3.8.6  | 1.18.5 | 1.4.1 | 3.19.3 | 20.1.0  |
+
+
+## Installing all prerequisites
+
+We do not include the optimized mathematical libraries in the instructions (BLAS, LAPACK,
+FFT library, ...) as these libraries should be standard on any optimized HPC system. 
+Also, the instructions below will need to be adapted to the specific 
+libraries that are being used.
+
+Other prerequisites: 
+  * libxc
+  * Python interpreter
+  * Python package NumPy
+  * Python package SciPy
+  * Python package ase
+
+
+### Installing libxc
+
+  * Installing libxc requires GNU automake and GNU buildtool besides GNU make and a 
+    C compiler. The build process is the usual GNU configure - make - make install 
+    cycle, but the `configure` script still needs to be generated with autoreconf.
+  * Download libxc:
+      * The latest version of libxc can be downloaded from 
+        [the libxc download page](https://www.tddft.org/programs/libxc/download/).
+        However, that version may not be officially supported by GPAW.
+      * It is also possible to download all recent versions of libxc from
+        [the libxc GitLab](https://gitlab.com/libxc/libxc)
+          * Select the tag corresponding to the version you want to download in the
+            branch/tag selection box.
+          * Then use the download button and select the desired file type.
+          * Dowload URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
+  * Untar the file in the build directory.
+  
+
+ 
+
+### Installing Python from scratch
+
+The easiest way to get Python on your system is to download an existing distribution
+(one will likely already be installed on your system). Python itself does have a lot 
+of dependencies though, definitely in its Standard Python Library. Many of the
+standard packages are never needed when executing the benchmark cases. Isolating them
+to compile a Python with minimal dependencies is beyond the scope though. We did
+compile Python without the necessary libraries for the standard libraries sqlite3
+and tkinter (the latter needing Tcl/Tk).
+
+Even though GPAW contains a lot of Python code, the Python interpreter is not the main
+performance-determining factor in the GPAW benchmark. Having a properly optimized installation
+of NumPy, SciPy and GPAW itself proves much more important.
+
+
+### Installing NumPy
+
+  * As NumPy relies on optimized libraries for its performance, one should carefully
+    select which NumPy package to download, or install NumPy from sources. How crucial 
+    this is, depends on the version of GPAW and the options selected when building 
+    GPAW.
+  * Given that GPAW also uses optimized libraries, it is generally advised to install
+    NumPy from sources instead to ensure that the same libraries are used as will be
+    used for GPAW to prevent conflicts between libraries that might otherwise occur.
+  * In most cases, NumPy will need a `site.cfg` file to point to the optimized libraries.
+    See the examples for various systems and the file `site.cfg.example` included in
+    the NumPy sources.
+
+
+### Installing SciPy
+
+  * Just as NumPy, SciPy relies on optimized libraries for its performance. It should
+    be installed after NumPy as it does get the information about which libraries to 
+    use from NumPy. Hence, when installing pre-built binaries, make sure they match
+    the NumPy binaries used.
+  * Just as is the case for NumPy, it may be better to install SciPy from sources.
+    [Instructions for installing SciPy from source can be found on the SciPy GitHub 
+    site](https://github.com/scipy/scipy/blob/master/INSTALL.rst.txt).
+
+
+### Installing ase
+
+  * Just as for any user-installed Python package, make sure you have created a
+    directory to install Python packages to and have added it to the front of PYTHONPATH.
+  * ase is [available on PyPi](https://pypi.org/project/ase/). It is also possible 
+    to [see a list of previous releases](https://pypi.org/project/ase/#history).
+  * The easiest way to install ase is using `pip` which will automatically download.
+    the requested version.
+
+
+## Configuring and installing GPAW
+
+### GPAW 1.5.2
+
+  * GPAW 1.5.2 uses `distutils`. Customization of the installation process is possible
+    through the `customize.py` file.
+  * The FFT library: According to the documentation, the following strategy is used
+      * The compile process searches (in this order) for ``libmkl_rt.so``,
+        ``libmkl_intel_lp64.so`` and ``libfftw3.so`.  First one found will be
+        loaded.
+      * If none is found, the built-in FFT from NumPy will be used. This does not need
+        to be a problem if NumPy provides a properly optimized FFT library.
+      * The choice can also be overwritten using the GPAW_FFTWSO environment variable.
+  * With certain compilers, the GPAW test suite produced crashes in `xc/xc.py`. The 
+    patch for GPAW 1.5.2 included in the [pathces](patches) subdirectory solved these
+    problems on the systems tested.
+
+
+### GPAW 19.8.1
+
+  * GPAW 19.8.1 uses `distutils`. Customization of the installation process is possible
+    through a `customize.py` file.
+  * The selection process of the FFT library has changed from version 1.5.2. It is 
+    now possible to specify the FFT library in `customize.py` or to simply select to
+    use the NumPy FFT routines.
+
+
+
+
+### GPAW 20.1.0 and 20.10.0
+
+  * GPAW 20.1.0 uses `setuptools`. Customization of the installation process is possible
+    through the `siteconfig.py` file. 
+  * The selection process of the FFT library is the same as in version 19.8.1, except 
+    that the settings are now in `siteconfrig.py` rather than `customize.py`.
+
+
+### All versions
+
+  * GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files 
+    can be found on the [GPAW website, Atomic PAW Setups page](https://wiki.fysik.dtu.dk/gpaw/setups/setups.html).
+    For the testing we used []`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
+    for all versions of GPAW. The easiest way to install these files is to simpy untar
+    the file and set the environment variable GPAW_SETUP_PATH to point to that directory.
+    In the examples provided we use the `share/gpaw-setups` subdirectory of the install
+    directory for this purpose.
+  * Up to and including version 20.1.0, GPAW does comes with a test suite which can be 
+    used after installation.
+      * Running the sequential tests:
+
+            gpaw test
+
+        Help is available through
+
+            gpaw test -h
+
+      * Running those tests, but using multiple cores (e.g., 4):
+
+            gpaw test -j 4
+
+        We did experience that crashed that cause segmentation faults get unnoticed 
+        in this setup. They are not mentioned as failed.
+
+      * Running the parallel benchmarks on a SLURM cluster will depend on the version of GPAW.
+
+          * Versions that build the parallel interpreter (19.8.1 and older):
+
+                srun -n 4 gpaw-python -m gpaw test
+
+          * Versions with the parallel so library using the regular Python interpreter (20.1.0 and above):
+
+                srun -n 4 python -m gpaw test
+
+
+      * Depending on the Python installation, some tests may fail with error messages that point
+        to a package in the Standard Python Library that is not present. Some of these errors have no
+        influence on the benchmarks as that part of the code is not triggered by the benchmark.
+  * The full test suite is missing in GPAW 20.10.0. There is a brief sequential test
+    that can be run with
+
+        gpaw test
+
+    and a parallel one that can be run with
+
+        gpaw -P 4 test
+
+  * Multiple versions of GPAW likely contain a bug in `c/bmgs/fd.c` (around line 44 
+    in GPAW 1.5.2). The code enforces vectorization on OpenMP 4 compilers by using
+    `#pragma omp simd`. However, it turns out that the data is not always correctly 
+    aligned, so if the reaction of the compiler to `#pragma omp simd` is to fully vectorize
+    and use load/store instructions for aligned data, crashes may occur. It did happen 
+    during the benchmark development when compiling with the Intel C compiler. The 
+    solution for that compiler is to add `-qno-openmp-simd` to the compiler flags.
+
+
+## Problems observed during testing
+
+  * On AMD Epyc systems, there seems to be a bug in the Intel MKL FFT libraries/FFTW 
+    wrappers in the 2020 compilers. Downgrading to the MKL libraries of the 2018 
+    compilers or using the FFTW libraries solves the problem.
+    This has been observed not only in GPAW, but also in some other DFT packages.
+  * The GPAW test code in versions 1.5.2 till 20.1.0 detects that matplotlib is not installed
+    and will skip this test. We did however observe a failed test when Python could not find
+    the SQLite package as the Python standard library sqlite3 package is used.

gpaw/build/examples/CalcUA-vaughan-rome/build_1.5.2_IntelPython3_icc.sh

+#!/bin/bash
+#
+# Installation script for GPAW 1.5.2:
+# * Using the existing IntelPython3 module on the system which has an optimized 
+#   NumPy and SciPy included.
+# * Using the matching version of ase, 3.17.0
+# * Compiling with the Intel compilers
+#
+# The FFT library is discovered at runtime. With the settings used in this script
+# this should be MKL FFT, but it is possible to change this at runtime to either 
+# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
+# (link below).
+#
+# The original installation instructions for GPAW can be found at
+# https://gitlab.com/gpaw/gpaw/-/blob/1.5.2/doc/install.rst
+#
+
+packageID='1.5.2-IntelPython3-icc'
+
+install_root=$VSC_SCRATCH/UEABS
+systemID=CalcUA-vaughan-rome
+
+download_dir=$install_root/Downloads
+install_dir=$install_root/$systemID/Packages/GPAW-manual/$packageID
+modules_dir=$install_root/$systemID/Modules/GPAW-manual
+build_dir="/dev/shm/$USER/GPAW-manual/$packageID"
+patch_dir=$VSC_DATA/Projects/PRACE/GPAW-experiments/UEABS/build/patches
+
+libxc_version='4.3.4'
+ase_version='3.17.0'
+GPAW_version='1.5.2'
+GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
+
+py_maj_min='3.7'
+
+################################################################################
+#
+# Prepare the system
+#
+
+#
+# Load modules
+#
+module purge
+module load calcua/2020a
+module load intel/2020a
+module load IntelPython3/2020a
+module load buildtools/2020a
+
+#
+# Create the directories and make sure they are clean if that matters
+#
+/usr/bin/mkdir -p $download_dir
+
+/usr/bin/mkdir -p $install_dir
+/usr/bin/rm -rf   $install_dir
+/usr/bin/mkdir -p $install_dir
+
+/usr/bin/mkdir -p $modules_dir
+
+/usr/bin/mkdir -p $build_dir
+/usr/bin/rm -rf   $build_dir
+/usr/bin/mkdir -p $build_dir
+
+
+
+################################################################################
+#
+# Download components
+#
+
+echo -e "\nDownloading files...\n"
+
+cd $download_dir
+
+# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
+libxc_file="libxc-$libxc_version.tar.bz2"
+libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
+[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
+
+# We do not download ase in this script. As it is pure python and doesn't need
+# customization, we will install it using pip right away.
+## https://files.pythonhosted.org/packages/d9/08/35969da23b641d3dfca46ba7559f651fcfdca81dbbc00b9058c934e75769/ase-3.17.0.tar.gz
+#ase_file="ase-$ase_version.tar.gz"
+#ase_url="https://files.pythonhosted.org/packages/d9/08/35969da23b641d3dfca46ba7559f651fcfdca81dbbc00b9058c934e75769"
+#[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
+
+# GPAW needs customization, so we need to download and unpack the sources.
+# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
+GPAW_file="gpaw-$GPAW_version.tar.gz"
+#GPAW_url="https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9"
+GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
+[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
+
+# Download GPAW-setup, a number of setup files for GPAW.
+# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
+GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
+GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
+[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\nInstalling libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $download_dir/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -fPIC"
+export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC"
+#export CFLAGS="-O0 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC"
+./configure --prefix="$install_dir" \
+            --disable-static --enable-shared --disable-fortran
+# Build
+make -j 16
+
+# Install
+make -j 16 install
+
+# Add bin, lib and include to the PATH variables
+PATH=$install_dir/bin:$PATH
+LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
+LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
+CPATH=$install_dir/include:$CPATH
+
+
+################################################################################
+#
+# Install ase
+#
+
+echo -e "\nInstalling ase...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+pip install --prefix=$install_dir --no-deps ase==$ase_version
+
+
+################################################################################
+#
+# Install GPAW-setups
+#
+
+echo -e "\nInstalling gpaw-setups...\n"
+
+mkdir -p $install_dir/share/gpaw-setups
+cd $install_dir/share/gpaw-setups
+tar -xf $download_dir/$GPAWsetups_file --strip-components=1
+
+
+################################################################################
+#
+# Install GPAW
+#
+
+echo -e "\nInstalling GPAW...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $download_dir/$GPAW_file
+
+# Apply patches
+patch -p0 <$patch_dir/gpaw-1.5.2.patch
+
+cd gpaw-$GPAW_version
+
+# Make the customize.py script
+mv customize.py customize.py.orig
+cat >customize.py <<EOF
+print( 'GPAW build INFO: Starting execution of the customization script' )
+print( 'GPAW build INFO: Variables at the start of the customization script' )
+print( 'GPAW build INFO: libraries =               ', libraries )
+print( 'GPAW build INFO: mpi_libaries =            ', mpi_libraries )
+print( 'GPAW build INFO: library_dirs =            ', library_dirs )
+print( 'GPAW build INFO: mpi_libary_dirs =         ', mpi_library_dirs )
+print( 'GPAW build INFO: runtime_library_dirs =    ', runtime_library_dirs )
+print( 'GPAW build INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
+print( 'GPAW build INFO: include_dirs =            ', include_dirs )
+print( 'GPAW build INFO: mpi_include_dirs =        ', mpi_include_dirs )
+print( 'GPAW build INFO: compiler =                ', compiler )
+print( 'GPAW build INFO: mpicompiler =             ', mpicompiler )
+print( 'GPAW build INFO: mpilinker =               ', mpilinker )
+print( 'GPAW build INFO: extra_compile_args =      ', extra_compile_args )
+print( 'GPAW build INFO: extra_link_args =         ', extra_link_args )
+print( 'GPAW build INFO: define_macros =           ', define_macros )
+print( 'GPAW build INFO: mpi_define_macros =       ', mpi_define_macros )
+print( 'GPAW build INFO: undef_macros =            ', undef_macros )
+print( 'GPAW build INFO: scalapack =               ', scalapack )
+print( 'GPAW build INFO: libvdwxc =                ', libvdwxc )
+print( 'GPAW build INFO: elpa =                    ', elpa )
+
+# Reset the lists of libraries as often the wrong BLAS library is picked up.
+libraries = []
+mpi_libraries = []
+
+# LibXC. Re-add the library (removed by resetting libraries).
+# There is no need to add the library directory as we do set library_dirs to the
+# content of LIBRARY_PATH further down.
+# There should be no need to add the include directory as it is in CPATH which is
+# set when we install gpaw. 
+#include_dirs.append('$install_dir/include')
+libraries.append('xc')
+
+# libvdwxc
+libvdwxc = False
+
+# ELPA
+elpa = False
+
+# ScaLAPACK
+scalapack = True
+mpi_libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
+mpi_define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
+mpi_define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
+
+# Add EasyBuild LAPACK/BLAS libs
+# This should also enable MKL FFTW according to the documentation of GPAW 1.5.2
+libraries += ['mkl_intel_lp64', 'mkl_sequential', 'mkl_core']
+
+# Add other EasyBuild library directories.
+library_dirs = os.environ['LIBRARY_PATH'].split(':')
+
+# Set the compilers
+compiler =    os.environ['CC']
+mpicompiler = os.environ['MPICC']
+mpilinker =   os.environ['MPICC']
+
+# We need extra_compile_args to have the right compiler options when re-compiling
+# files for gpaw-python. It does imply double compiler options for the other
+# compiles though.
+extra_compile_args = os.environ['CFLAGS'].split(' ')
+
+print( 'GPAW build INFO: Variables at the end of the customization script' )
+print( 'GPAW build INFO: libraries =               ', libraries )
+print( 'GPAW build INFO: mpi_libaries =            ', mpi_libraries )
+print( 'GPAW build INFO: library_dirs =            ', library_dirs )
+print( 'GPAW build INFO: mpi_libary_dirs =         ', mpi_library_dirs )
+print( 'GPAW build INFO: runtime_library_dirs =    ', runtime_library_dirs )
+print( 'GPAW build INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
+print( 'GPAW build INFO: include_dirs =            ', include_dirs )
+print( 'GPAW build INFO: mpi_include_dirs =        ', mpi_include_dirs )
+print( 'GPAW build INFO: compiler =                ', compiler )
+print( 'GPAW build INFO: mpicompiler =             ', mpicompiler )
+print( 'GPAW build INFO: mpilinker =               ', mpilinker )
+print( 'GPAW build INFO: extra_compile_args =      ', extra_compile_args )
+print( 'GPAW build INFO: extra_link_args =         ', extra_link_args )
+print( 'GPAW build INFO: define_macros =           ', define_macros )
+print( 'GPAW build INFO: mpi_define_macros =       ', mpi_define_macros )
+print( 'GPAW build INFO: undef_macros =            ', undef_macros )
+print( 'GPAW build INFO: scalapack =               ', scalapack )
+print( 'GPAW build INFO: libvdwxc =                ', libvdwxc )
+print( 'GPAW build INFO: elpa =                    ', elpa )
+print( 'GPAW build INFO: Ending execution of the customization script' )
+EOF
+
+# Now install gpaw
+export CC=icc
+export MPICC=mpiicc
+export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -qno-openmp-simd"
+export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -qno-openmp-simd"
+#export CFLAGS="-O0 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -qno-openmp-simd"
+python setup.py build -j 32
+
+# Install GPAW
+python setup.py install --prefix="$install_dir"
+
+
+################################################################################
+#
+# Finish the install
+#
+
+echo -e "\nCleaning up and making the LUA-module GPAW-manual/$packageID...\n"
+
+
+# Go to a different directory before cleaning up the build directory
+cd $modules_dir
+#/bin/rm -rf $build_dir
+
+# Create a module file
+cat >$packageID.lua <<EOF
+help([==[
+
+Description
+===========
+
+GPAW $GPAW_version for the UEABS benchmark.
+
+Configuration:
+  * IntelPython3/2020a module, also for NumPy and SciPy
+  * Parallel GPAW $GPAW_version with ase $ase_version
+  * FFT library selected at runtime. The default with the path
+    as set through this module should be MKL but it can be changed
+    by setting GPAW_FFTWSO as indicated in the install instructions at
+    https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/install.rst
+  * libxc and GPAW compiled with the Intel compilers
+
+
+More information
+================
+ - Homepage: http://wiki.fysik.dtu.dk/gpaw
+ - Documentation:
+    - GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
+    - Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
+    - ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
+
+
+Included extensions
+===================
+ase-$ase_version, gpaw-$GPAW_version
+]==])
+
+whatis([==[Description: GPAW $GPAW_version with ase $ase_version: UEABS benchmark configuration.]==])
+
+conflict("GPAW")
+conflict("GPAW-manual")
+
+if not ( isloaded("calcua/2020a") ) then
+    load("calcua/2020a")
+end
+
+if not ( isloaded("intel/2020a") ) then
+    load("intel/2020a")
+end
+
+if not ( isloaded("IntelPython3/2020a") ) then
+    load("IntelPython3/2020a")
+end
+
+
+
+prepend_path("PATH",            "$install_dir/bin")
+prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
+prepend_path("LIBRARY_PATH",    "$install_dir/lib")
+prepend_path("PYTHONPATH",      "$install_dir/lib/python$py_maj_min/site-packages")
+
+setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
+EOF

gpaw/build/examples/CalcUA-vaughan-rome/build_1.5.2_Python37icc_icc.sh

+#!/bin/bash
+#
+# Installation script for GPAW 1.5.2:
+# * Using the existing IntelPython3 module on the system which has an optimized 
+#   NumPy and SciPy included.
+# * Using the matching version of ase, 3.17.0
+# * Compiling with the Intel compilers
+#
+# The FFT library is discovered at runtime. With the settings used in this script
+# this should be MKL FFT, but it is possible to change this at runtime to either 
+# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
+# (link below).
+#
+# The original installation instructions for GPAW can be found at
+# https://gitlab.com/gpaw/gpaw/-/blob/1.5.2/doc/install.rst
+#
+
+packageID='1.5.2-Python37icc-icc'
+
+install_root=$VSC_SCRATCH/UEABS
+systemID=CalcUA-vaughan-rome
+
+download_dir=$install_root/Downloads
+install_dir=$install_root/$systemID/Packages/GPAW-manual/$packageID
+modules_dir=$install_root/$systemID/Modules/GPAW-manual
+build_dir="/dev/shm/$USER/GPAW-manual/$packageID"
+patch_dir=$VSC_DATA/Projects/PRACE/GPAW-experiments/UEABS/build/patches
+
+libxc_version='4.3.4'
+cython_version='0.29.21'
+numpy_version='1.18.5'
+#numpy_version='1.19.2'
+scipy_version='1.4.1'
+#scipy_version='1.5.3'
+ase_version='3.17.0'
+GPAW_version='1.5.2'
+GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
+
+py_maj_min='3.7'
+
+################################################################################
+#
+# Prepare the system
+#
+
+#
+# Load modules
+#
+module purge
+module load calcua/2020a
+module load personal/UEABS
+module load intel/2020a
+module load buildtools/2020a
+module load FFTW/3.3.8-intel-2020a
+module load Python/3.7.9-intel-2020a-Cfg5-UEABS-GPAW
+
+#
+# Create the directories and make sure they are clean if that matters
+#
+/usr/bin/mkdir -p $download_dir
+
+/usr/bin/mkdir -p $install_dir
+/usr/bin/rm -rf   $install_dir
+/usr/bin/mkdir -p $install_dir
+
+/usr/bin/mkdir -p $modules_dir
+
+/usr/bin/mkdir -p $build_dir
+/usr/bin/rm -rf   $build_dir
+/usr/bin/mkdir -p $build_dir
+
+
+
+################################################################################
+#
+# Download components
+#
+
+echo -e "\nDownloading files...\n"
+
+cd $download_dir
+
+# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
+libxc_file="libxc-$libxc_version.tar.bz2"
+libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
+[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
+
+# We do not download Cython but simply install it using pip.
+
+# NumPy needs customizations, so we need to download and unpack the sources
+numpy_file="numpy-$numpy_version.zip"
+numpy_url="https://pypi.python.org/packages/source/n/numpy"
+[[ -f $numpy_file ]] || wget "$numpy_url/$numpy_file"
+
+# SciPy
+scipy_file="scipy-$scipy_version.tar.gz"
+scipy_url="https://pypi.python.org/packages/source/s/scipy"
+[[ -f $scipy_file ]] || wget "$scipy_url/$scipy_file"
+
+# We do not download ase in this script. As it is pure python and doesn't need
+# customization, we will install it using pip right away.
+## https://files.pythonhosted.org/packages/d9/08/35969da23b641d3dfca46ba7559f651fcfdca81dbbc00b9058c934e75769/ase-3.17.0.tar.gz
+#ase_file="ase-$ase_version.tar.gz"
+#ase_url="https://files.pythonhosted.org/packages/d9/08/35969da23b641d3dfca46ba7559f651fcfdca81dbbc00b9058c934e75769"
+#[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
+
+# GPAW needs customization, so we need to download and unpack the sources.
+# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
+GPAW_file="gpaw-$GPAW_version.tar.gz"
+GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
+[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
+
+# Download GPAW-setup, a number of setup files for GPAW.
+# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
+GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
+GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
+[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
+
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\nInstalling libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $download_dir/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+#export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -fPIC"
+export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC"
+#export CFLAGS="-O0 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC"
+./configure --prefix="$install_dir" \
+            --disable-static --enable-shared --disable-fortran
+# Build
+make -j 16
+
+# Install
+make -j 16 install
+
+# Add bin, lib and include to the PATH variables
+PATH=$install_dir/bin:$PATH
+LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
+LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
+CPATH=$install_dir/include:$CPATH
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\nInstalling Cython...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+cd $build_dir
+
+pip3 install --prefix=$install_dir --no-deps cython==$cython_version
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\nInstalling NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $download_dir/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <<EOF
+[DEFAULT]
+library_dirs = $EBROOTFFTW/lib:$MKLROOT/lib/intel64_lin
+include_dirs = $EBROOTFFTW/include:$MKLROOT/include
+search_static_first=True
+[mkl]
+lapack_libs = mkl_rt
+mkl_libs = mkl_rt
+[fftw]
+libraries = fftw3
+EOF
+
+export CC=icc
+export CFLAGS='-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fwrapv -fPIC -diag-disable=1678,10148,111,188,3438,2650,3175,1890'
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS='-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC'
+export LD=icc
+
+# Build NumPy.
+# Note: Try python setup.py build --help-compiler and python setup.py build --help-fcompiler to list the
+# choice of compilers. We did experience though that if the Intel compilers are selected explicitly,
+# the wrong linker was used leading to problems there.
+python setup.py build -j 32
+
+# Install NumPy
+python setup.py install --prefix $install_dir
+
+# Brief test: Should not be run in the NumPy build directory...
+cd $build_dir
+python -c "import numpy"
+
+
+################################################################################
+#
+# Install SciPy
+#
+
+echo -e "\nInstalling SciPy...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $download_dir/$scipy_file
+
+cd scipy-$scipy_version
+
+export CC=icc
+export CFLAGS='-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fwrapv -fPIC -diag-disable=1678,10148,111,188,3438'
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+# FFLAGS is used in the fitpack Makefile
+export FFLAGS='-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC'
+# FOPT is used in the odrpack Makefile
+export FOPT='-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC'
+export LD=icc
+
+# Build scipy
+python setup.py build -j 32 
+
+# Install scipy
+python setup.py install --prefix $install_dir
+
+# Brief test: Should not be run in the NumPy build directory...
+cd $build_dir
+python -c "import scipy"
+
+
+################################################################################
+#
+# Install ase
+#
+
+echo -e "\nInstalling ase...\n"
+
+cd $build_dir
+
+pip3 install --prefix=$install_dir --no-deps ase==$ase_version
+
+
+################################################################################
+#
+# Install GPAW-setups
+#
+
+echo -e "\nInstalling gpaw-setups...\n"
+
+mkdir -p $install_dir/share/gpaw-setups
+cd $install_dir/share/gpaw-setups
+tar -xf $download_dir/$GPAWsetups_file --strip-components=1
+
+
+################################################################################
+#
+# Install GPAW
+#
+
+echo -e "\nInstalling GPAW...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $download_dir/$GPAW_file
+
+# Apply patches
+patch -p0 <$patch_dir/gpaw-1.5.2.patch
+
+cd gpaw-$GPAW_version
+
+# Make the customize.py script
+mv customize.py customize.py.orig
+cat >customize.py <<EOF
+print( 'GPAW build INFO: Starting execution of the customization script' )
+print( 'GPAW build INFO: Variables at the start of the customization script' )
+print( 'GPAW build INFO: libraries =               ', libraries )
+print( 'GPAW build INFO: mpi_libaries =            ', mpi_libraries )
+print( 'GPAW build INFO: library_dirs =            ', library_dirs )
+print( 'GPAW build INFO: mpi_libary_dirs =         ', mpi_library_dirs )
+print( 'GPAW build INFO: runtime_library_dirs =    ', runtime_library_dirs )
+print( 'GPAW build INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
+print( 'GPAW build INFO: include_dirs =            ', include_dirs )
+print( 'GPAW build INFO: mpi_include_dirs =        ', mpi_include_dirs )
+print( 'GPAW build INFO: compiler =                ', compiler )
+print( 'GPAW build INFO: mpicompiler =             ', mpicompiler )
+print( 'GPAW build INFO: mpilinker =               ', mpilinker )
+print( 'GPAW build INFO: extra_compile_args =      ', extra_compile_args )
+print( 'GPAW build INFO: extra_link_args =         ', extra_link_args )
+print( 'GPAW build INFO: define_macros =           ', define_macros )
+print( 'GPAW build INFO: mpi_define_macros =       ', mpi_define_macros )
+print( 'GPAW build INFO: undef_macros =            ', undef_macros )
+print( 'GPAW build INFO: scalapack =               ', scalapack )
+print( 'GPAW build INFO: libvdwxc =                ', libvdwxc )
+print( 'GPAW build INFO: elpa =                    ', elpa )
+
+# Reset the lists of libraries as often the wrong BLAS library is picked up.
+libraries = []
+mpi_libraries = []
+
+# LibXC. Re-add the library (removed by resetting libraries).
+# There is no need to add the library directory as we do set library_dirs to the
+# content of LIBRARY_PATH further down.
+# There should be no need to add the include directory as it is in CPATH which is
+# set when we install gpaw. 
+#include_dirs.append('$install_dir/include')
+libraries.append('xc')
+
+# libvdwxc
+libvdwxc = False
+
+# ELPA
+elpa = False
+
+# ScaLAPACK
+scalapack = True
+mpi_libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
+mpi_define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
+mpi_define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
+
+# Add EasyBuild LAPACK/BLAS libs
+# This should also enable MKL FFTW according to the documentation of GPAW 1.5.2
+libraries += ['mkl_intel_lp64', 'mkl_sequential', 'mkl_core']
+
+# Add other EasyBuild library directories.
+library_dirs = os.environ['LIBRARY_PATH'].split(':')
+
+# Set the compilers
+compiler =    os.environ['CC']
+mpicompiler = os.environ['MPICC']
+mpilinker =   os.environ['MPICC']
+
+# We need extra_compile_args to have the right compiler options when re-compiling
+# files for gpaw-python. It does imply double compiler options for the other
+# compiles though.
+extra_compile_args = os.environ['CFLAGS'].split(' ')
+
+print( 'GPAW build INFO: Variables at the end of the customization script' )
+print( 'GPAW build INFO: libraries =               ', libraries )
+print( 'GPAW build INFO: mpi_libaries =            ', mpi_libraries )
+print( 'GPAW build INFO: library_dirs =            ', library_dirs )
+print( 'GPAW build INFO: mpi_libary_dirs =         ', mpi_library_dirs )
+print( 'GPAW build INFO: runtime_library_dirs =    ', runtime_library_dirs )
+print( 'GPAW build INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
+print( 'GPAW build INFO: include_dirs =            ', include_dirs )
+print( 'GPAW build INFO: mpi_include_dirs =        ', mpi_include_dirs )
+print( 'GPAW build INFO: compiler =                ', compiler )
+print( 'GPAW build INFO: mpicompiler =             ', mpicompiler )
+print( 'GPAW build INFO: mpilinker =               ', mpilinker )
+print( 'GPAW build INFO: extra_compile_args =      ', extra_compile_args )
+print( 'GPAW build INFO: extra_link_args =         ', extra_link_args )
+print( 'GPAW build INFO: define_macros =           ', define_macros )
+print( 'GPAW build INFO: mpi_define_macros =       ', mpi_define_macros )
+print( 'GPAW build INFO: undef_macros =            ', undef_macros )
+print( 'GPAW build INFO: scalapack =               ', scalapack )
+print( 'GPAW build INFO: libvdwxc =                ', libvdwxc )
+print( 'GPAW build INFO: elpa =                    ', elpa )
+print( 'GPAW build INFO: Ending execution of the customization script' )
+EOF
+
+# Build gpaw
+export CC=icc
+export MPICC=mpiicc
+#export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -qno-openmp-simd"
+export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -qno-openmp-simd"
+#export CFLAGS="-O0 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -qno-openmp-simd"
+python setup.py build -j 32
+
+# Install GPAW
+python setup.py install --prefix="$install_dir"
+
+
+################################################################################
+#
+# Finish the install
+#
+
+echo -e "\nCleaning up and making the LUA-module GPAW-manual/$packageID...\n"
+
+
+# Go to a different directory before cleaning up the build directory
+cd $modules_dir
+#/bin/rm -rf $build_dir
+
+# Create a module file
+cat >$packageID.lua <<EOF
+help([==[
+
+Description
+===========
+
+GPAW $GPAW_version for the UEABS benchmark.
+
+Configuration:
+  * Python 3 compiled with the Intel compilers pre-installed on the system
+  * NumPy/SciPy using MKL for linear algebra and FFTW for FFT.
+  * Parallel GPAW $GPAW_version with ase $ase_version
+  * FFT library selected at runtime. The default with the path
+    as set through this module should be MKL. However, we set GPAW_FFTWSO
+    in this module to select the regular FFTW libraries as we have 
+    experienced problems with the Intel MKL FFTW routines on some systems.
+    Setting GPAW_FFTWSO is documented in the install instructions at
+    https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/install.rst
+  * libxc and GPAW compiled with the Intel compilers
+
+
+More information
+================
+ - Homepage: http://wiki.fysik.dtu.dk/gpaw
+ - Documentation:
+    - GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
+    - Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
+    - ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
+
+
+Included extensions
+===================
+ase-$ase_version, gpaw-$GPAW_version
+]==])
+
+whatis([==[Description: GPAW $GPAW_version with ase $ase_version: UEABS benchmark configuration.]==])
+
+conflict("GPAW")
+conflict("GPAW-manual")
+
+if not ( isloaded("calcua/2020a") ) then
+    load("calcua/2020a")
+end
+
+-- if not ( isloaded("personal/UEABS") ) then
+--     load("personal/UEABS")
+-- end
+
+if not ( isloaded("intel/2020a") ) then
+    load("intel/2020a")
+end
+
+if not ( isloaded("FFTW/3.3.8-intel-2020a") ) then
+    load("FFTW/3.3.8-intel-2020a")
+end
+
+if not ( isloaded("Python/3.7.9-intel-2020a-Cfg5-UEABS-GPAW") ) then
+    load("Python/3.7.9-intel-2020a-Cfg5-UEABS-GPAW")
+end
+
+prepend_path("PATH",            "$install_dir/bin")
+prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
+prepend_path("LIBRARY_PATH",    "$install_dir/lib")
+prepend_path("PYTHONPATH",      "$install_dir/lib/python$py_maj_min/site-packages")
+
+setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
+setenv("GPAW_FFTWSO",     "libfftw3.so")
+EOF

gpaw/build/patches/gpaw-1.5.2.patch

+Patch written by Kurt Lust, UAntwerp for PRACE IP UEABS
+ - Backports a change to c/xc/tpss.c from 19.8.1 to 1.5.2.
+ - Backports a change to c/xc/revtpss.c from 19.8.1 to 1.5.2.
+--- gpaw-1.5.2/c/xc/tpss.c.orig	2020-10-22 16:15:29.000000000 +0200
++++ gpaw-1.5.2/c/xc/tpss.c	2020-10-22 16:37:41.000000000 +0200
+@@ -427,12 +427,11 @@
+   int nder = (nspin==XC_UNPOLARIZED) ? 1 : 3;
+   for(i=0; i<nder; i++){
+     if(nspin==XC_UNPOLARIZED) dauxdsigma[i] /= 2.;
+-    double dCdsigma[i];
+-    dCdsigma[i]=  dCdcsi*dcsidsigma[i];
++    double dCdsigma = dCdcsi*dcsidsigma[i];
+ 
+     /* partial derivatives*/
+-    de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma[i] * zsq
+-      - zsq * (dens * dCdsigma[i] * aux + (1.0 + C) * dauxdsigma[i]);
++    de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma * zsq
++      - zsq * (dens * dCdsigma * aux + (1.0 + C) * dauxdsigma[i]);
+ 
+   }
+ }
+--- gpaw-1.5.2/c/xc/revtpss.c.orig	2020-10-22 16:15:29.000000000 +0200
++++ gpaw-1.5.2/c/xc/revtpss.c	2020-10-22 17:12:40.000000000 +0200
+@@ -231,12 +231,11 @@
+   int nder = (p->common.nspin==XC_UNPOLARIZED) ? 1 : 3;
+   for(i=0; i<nder; i++){
+     if(p->common.nspin==XC_UNPOLARIZED) dauxdsigma[i] /= 2.;
+-    double dCdsigma[i]; 
+-    dCdsigma[i]=  dCdcsi*dcsidsigma[i];
+-	
++    double dCdsigma =  dCdcsi*dcsidsigma[i];
++
+     /* partial derivatives*/
+-    de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma[i] * zsq
+-      - zsq * (dens * dCdsigma[i] * aux + (1.0 + C) * dauxdsigma[i]);
++    de_PKZBdsigma[i] = de_PBEdsigma[i] * (1.0 + C * zsq) + dens * e_PBE * dCdsigma * zsq
++      - zsq * (dens * dCdsigma * aux + (1.0 + C) * dauxdsigma[i]);
+ 
+   }
+ } 

nemo/README.md

+
 # NEMO
 
 
 ## Summary Version
 
-1.0
+1.1
 
 ## Purpose of Benchmark
 
@@ -35,90 +36,92 @@ The model is implemented in Fortran 90, with pre-processing (C-pre-processor). I
     ```
     ./make_xios --arch local
     ```
+    Files for the PRACE Tier-0 systems are available under [architecture_files](architecture_files) folder.
 
-Note that XIOS requires `Netcdf4`. Please load the appropriate `HDF5` and `NetCDF4` modules. You might have to change the path in the configuration file.
+Note that XIOS requires `Netcdf4`. Please load the appropriate `HDF5` and `NetCDF4` modules. If path to these models are not loaded, you might have to change the path in the configuration file.
 
 ### Building NEMO
 1.	Download the XIOS source code:
 	```
     svn co https://forge.ipsl.jussieu.fr/nemo/svn/NEMO/releases/release-4.0
     ```
-2.	Copy and setup the appropriate architecture file in the arch folder. The following changes are recommended:
+2.	Copy and setup the appropriate architecture file in the arch folder. Files for the PRACE Tier-0 systems are available under [architecture_files](architecture_files) folder. The following changes are recommended for the GNU compilers:
     ```
     a.	add the `-lnetcdff` and `-lstdc++` flags to NetCDF flags
     b.	using `mpif90` which is a MPI binding of `gfortran-4.9`
     c.	add `-cpp` and `-ffree-line-length-none` to Fortran flags
-    d.	swap out `gmake` with `make`
-    ```
-
-3.	Then build the executable with the following command
-	```
-    ./makenemo -m MY_CONFIG -r GYRE_XIOS -n MY_GYRE add_key "key_nosignedzero"
     ```
-4.	Apply the patch as described here to measure step time :
+3.	Apply the patch as described here to measure step time :
     ```
     https://software.intel.com/en-us/articles/building-and-running-nemo-on-xeon-processors
     ```
+    You may also use [nemogcm.F90](nemogcm.F90)  by replacing it with `src/OCE/nemogcm.F90`
+    
+4.  go to `cfgs` folder and add `GYRE_testing OCE TOP` line to `refs_cfg.txt` file.
+    Then:
+    ```
+    mkdir GYRE_testing
+	rsync -arv GYRE_PISCES/* GYRE_testing/
+	mv GYRE_testing/cpp_GYRE_PISCES.fcm GYRE_testing/cpp_GYRE_testing.fcm
+	In GYRE_testing/cpp_GYRE_testing.fcm file replace key_top with key_nosignedzero
+    ```
+
+5.	Build the executable with the following command
+	```
+    ../makenemo -m MY_CONFIG -r GYRE_testing
+    ```
 
 ## Mechanics of Running Benchmark
 
 ### Prepare input files
-	cd MY_GYRE/EXP00
+	cd GYRE_testing/EXP00
 	sed -i '/using_server/s/false/true/' iodef.xml
-	sed -i '/&nameos/a ln_useCT = .false.' namelist_cfg
-	sed -i '/&namctl/a nn_bench = 1' namelist_cfg
+	sed -i '/ln_bench/s/false/true/' namelist_cfg
 
 ### Run the experiment interactively
-	mpirun -n 4 ../BLD/bin/nemo.exe -n 2 $PATH_TO_XIOS/bin/xios_server.exe
+	mpirun -n 4 nemo : -n 2 $PATH_TO_XIOS/bin/xios_server.exe
 
 ### GYRE configuration with higher resolution
 Modify configuration (for example for the test case A):
 ```
-    rm -f time.step solver.stat output.namelist.dyn ocean.output  slurm-*  GYRE_* mesh_mask_00*
-    jp_cfg=4
+    rm -f time.step solver.stat output.namelist.dyn ocean.output  slurm-*  GYRE_*
     sed -i -r \
-        -e 's/^( *nn_itend *=).*/\1 21600/' \
-        -e 's/^( *nn_stock *=).*/\1 21600/' \
-        -e 's/^( *nn_write *=).*/\1 1000/' \
-        -e 's/^( *jp_cfg *=).*/\1 '"$jp_cfg"'/' \
-        -e 's/^( *jpidta *=).*/\1 '"$(( 30 * jp_cfg +2))"'/' \
-        -e 's/^( *jpjdta *=).*/\1 '"$(( 20 * jp_cfg +2))"'/' \
-        -e 's/^( *jpiglo *=).*/\1 '"$(( 30 * jp_cfg +2))"'/' \
-        -e 's/^( *jpjglo *=).*/\1 '"$(( 20 * jp_cfg +2))"'/' \
+        -e 's/^( *nn_itend *=).*/\1 101/' \
+        -e 's/^( *nn_write *=).*/\1 4320/' \
+        -e 's/^( *nn_GYRE *=).*/\1 48/' \
+        -e 's/^( *rn_rdt *=).*/\1 1200/' \
         namelist_cfg
 ```
 
 ## Verification of Results
-The GYRE configuration is set through the `namelist_cfg` file. The horizontal resolution is determined by setting `jp_cfg` as follows:
+The GYRE configuration is set through the `namelist_cfg` file. The horizontal resolution is determined by setting `nn_GYRE` as follows:
     
    ```
-   Jpiglo = 30 × jp_cfg + 2
-   Jpjglo = 20 × jp_cfg + 2
+   Jpiglo = 30 × nn_GYRE + 2
+   Jpjglo = 20 × nn_GYRE + 2
    ```
 
-In this configuration, we use a default value of 30 ocean levels, depicted by `jpk=31`. The GYRE configuration is an ideal case for benchmark tests as it is very simple to increase the resolution and perform both weak and strong scalability experiment using the same input files.  We use two configurations as follows:
+In this configuration, we use a default value of 30 ocean levels, depicted by `jpkglo=31`. The GYRE configuration is an ideal case for benchmark tests as it is very simple to increase the resolution and perform both weak and strong scalability experiment using the same input files.  We use two configurations as follows:
 
 Test Case A:
 ```
-    jp_cfg = 128 suitable up to 1000 cores
-	Number of Days: 20
-	Number of Time steps: 1440
+    nn_GYRE = 48 suitable up to 1000 cores
+	Number of Time steps: 101
 	Time step size: 20 mins
 	Number of seconds per time step: 1200
 ```
 Test Case B:
 ```
-    jp_cfg = 256 suitable up to 20,000 cores.
-	Number of Days (real): 80
-	Number of time step: 4320
+    nn_GYRE = 192 suitable up to 20,000 cores.
+	Number of time step: 101
 	Time step size(real): 20 mins
 	Number of seconds per time step: 1200
 ```
 
 We performed scalability test on 512 cores and 1024 cores for test case A. We performed scalability test for 4096 cores, 8192 cores and 16384 cores for test case B.
-
 Both these test cases can give us quite good understanding of node performance and interconnect behavior. 
-We switch off the generation of mesh files by setting the `flag nn_mesh = 0` in the `namelist_ref` file. Also `using_server = false` is defined in `io_server` file.
+<!--We switch off the generation of mesh files by setting the `flag nn_mesh = 0` in the `namelist_ref` file. 
+Also `using_server = false` is defined in `io_server` file.-->
 
 We report the performance in step time which is the total computational time averaged over the number of time steps for different test cases. 
 This helps us to compare systems in a standard manner across all combinations of system architectures. 

nemo/architecture_files/NEMO/arch-JUWELS.fcm

+# generic ifort compiler options for JUWELS
+#
+# NCDF_HOME   root directory containing lib and include subdirectories for netcdf4
+# HDF5_HOME   root directory containing lib and include subdirectories for HDF5
+# XIOS_HOME   root directory containing lib for XIOS
+# OASIS_HOME  root directory containing lib for OASIS
+#
+# NCDF_INC    netcdf4 include file
+# NCDF_LIB    netcdf4 library
+# XIOS_INC    xios include file    (taken into accound only if key_iomput is activated)
+# XIOS_LIB    xios library         (taken into accound only if key_iomput is activated)
+# OASIS_INC   oasis include file   (taken into accound only if key_oasis3 is activated)
+# OASIS_LIB   oasis library        (taken into accound only if key_oasis3 is activated)
+#
+# FC          Fortran compiler command
+# FCFLAGS     Fortran compiler flags
+# FFLAGS      Fortran 77 compiler flags
+# LD          linker
+# LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries
+# FPPFLAGS    pre-processing flags
+# AR          assembler
+# ARFLAGS     assembler flags
+# MK          make
+# USER_INC    complete list of include files
+# USER_LIB    complete list of libraries to pass to the linker
+# CC          C compiler used to compile conv for AGRIF
+# CFLAGS      compiler flags used with CC
+#
+# Note that:
+#  - unix variables "$..." are accpeted and will be evaluated before calling fcm.
+#  - fcm variables are starting with a % (and not a $)
+#
+
+%NCDF_HOME           /gpfs/software/juwels/stages/2019a/software/netCDF/4.6.3-ipsmpi-2019a.1/
+%NCDF_HOME2          /gpfs/software/juwels/stages/2019a/software/netCDF-Fortran/4.4.5-ipsmpi-2019a.1/
+%HDF5_HOME           /gpfs/software/juwels/stages/2019a/software/HDF5/1.10.5-ipsmpi-2019a.1/
+%CURL                /gpfs/software/juwels/stages/2019a/software/cURL/7.64.1-GCCcore-8.3.0/lib/
+
+%XIOS_HOME           /p/project/prpb86/nemo2/xios-2.5/
+%OASIS_HOME          /not/defined
+
+%HDF5_LIB            -L%HDF5_HOME/lib -L%CURL -lhdf5_hl -lhdf5
+%GCCLIB
+
+%NCDF_INC            -I%NCDF_HOME/include -I%NCDF_HOME2/include -I%HDF5_HOME/include
+%NCDF_LIB            -L%NCDF_HOME/lib %HDF5_LIB -L%CURL -L%NCDF_HOME2/lib -lnetcdff -lnetcdf -L%GCCLIB -lstdc++ -lz -lcurl -lgpfs
+
+%XIOS_INC            -I%XIOS_HOME/inc
+%XIOS_LIB            -L%XIOS_HOME/lib -lxios -L%GCCLIB -lstdc++
+
+%OASIS_INC           -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
+%OASIS_LIB           -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip
+
+%CPP                 icc -E 
+%FC                  mpifort
+%FCFLAGS             -O3 -r8 -funroll-all-loops -traceback
+ 
+%FFLAGS              %FCFLAGS
+%LD                  mpifort
+%LDFLAGS             -lstdc++ -lifcore -O3 -traceback
+%FPPFLAGS            -P -C -traditional 
+
+%AR                  ar
+%ARFLAGS             -r
+
+%MK                  make
+%USER_INC            %XIOS_INC %OASIS_INC %NCDF_INC
+%USER_LIB            %XIOS_LIB %OASIS_LIB %NCDF_LIB
+
+%CC                  cc
+%CFLAGS              -O0

nemo/architecture_files/NEMO/arch-M100.fcm

+# generic gfortran compiler options for linux M100
+#
+# NCDF_HOME   root directory containing lib and include subdirectories for netcdf4
+# HDF5_HOME   root directory containing lib and include subdirectories for HDF5
+# XIOS_HOME   root directory containing lib for XIOS
+# OASIS_HOME  root directory containing lib for OASIS
+#
+# NCDF_INC    netcdf4 include file
+# NCDF_LIB    netcdf4 library
+# XIOS_INC    xios include file    (taken into accound only if key_iomput is activated)
+# XIOS_LIB    xios library         (taken into accound only if key_iomput is activated)
+# OASIS_INC   oasis include file   (taken into accound only if key_oasis3 is activated)
+# OASIS_LIB   oasis library        (taken into accound only if key_oasis3 is activated)
+#
+# FC          Fortran compiler command
+# FCFLAGS     Fortran compiler flags
+# FFLAGS      Fortran 77 compiler flags
+# LD          linker
+# LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries
+# FPPFLAGS    pre-processing flags
+# AR          assembler
+# ARFLAGS     assembler flags
+# MK          make
+# USER_INC    complete list of include files
+# USER_LIB    complete list of libraries to pass to the linker
+# CC          C compiler used to compile conv for AGRIF
+# CFLAGS      compiler flags used with CC
+#
+# Note that:
+#  - unix variables "$..." are accpeted and will be evaluated before calling fcm.
+#  - fcm variables are starting with a % (and not a $)
+#
+
+%NCDF_HOME           /cineca/prod/opt/libraries/netcdf/4.7.3/gnu--8.4.0/
+%NCDF_HOME2          /cineca/prod/opt/libraries/netcdff/4.5.2/gnu--8.4.0/
+%HDF5_HOME           /cineca/prod/opt/libraries/hdf5/1.12.0/gnu--8.4.0/
+%XIOS_HOME           /m100_work/Ppp4x_5387/xios-2.5/
+%OASIS_HOME          /not/defined
+
+
+%HDF5_LIB            -L%HDF5_HOME/lib -lhdf5_hl -lhdf5
+%GCCLIB              /cineca/prod/opt/compilers/gnu/8.4.0/none/lib64/
+
+%NCDF_INC            -I%NCDF_HOME/include -I%NCDF_HOME2/include -I%HDF5_HOME/include
+%NCDF_LIB            -L%NCDF_HOME/lib %HDF5_LIB -L%NCDF_HOME2/lib -lnetcdff -lnetcdf -L%GCCLIB -lstdc++ -lz -lcurl -lgpfs
+%XIOS_INC            -I%XIOS_HOME/inc
+%XIOS_LIB            -L%XIOS_HOME/lib -lxios -L%GCCLIB -lstdc++
+
+%OASIS_INC           -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
+%OASIS_LIB           -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip
+
+%CPP                 cpp -Dkey_nosignedzero
+%FC                  mpif90 
+%FCFLAGS             -fdefault-real-8 -fno-second-underscore -O3 -funroll-all-loops -fcray-pointer -cpp -ffree-line-length-none -Dgfortran
+%FFLAGS              %FCFLAGS
+%LD                  %FC
+%LDFLAGS
+%FPPFLAGS            -P -C -traditional -x f77-cpp-input
+
+%AR                  ar
+%ARFLAGS             rs
+%MK                  make
+%USER_INC            %XIOS_INC %OASIS_INC %NCDF_INC 
+%USER_LIB            %XIOS_LIB %OASIS_LIB %NCDF_LIB 
+
+
+
+%CC                  cc
+%CFLAGS              -O0

nemo/architecture_files/NEMO/arch-SuperMUC.fcm

+# generic ifort compiler options for SuperMUC
+#
+# NCDF_HOME   root directory containing lib and include subdirectories for netcdf4
+# HDF5_HOME   root directory containing lib and include subdirectories for HDF5
+# XIOS_HOME   root directory containing lib for XIOS
+# OASIS_HOME  root directory containing lib for OASIS
+#
+# NCDF_INC    netcdf4 include file
+# NCDF_LIB    netcdf4 library
+# XIOS_INC    xios include file    (taken into accound only if key_iomput is activated)
+# XIOS_LIB    xios library         (taken into accound only if key_iomput is activated)
+# OASIS_INC   oasis include file   (taken into accound only if key_oasis3 is activated)
+# OASIS_LIB   oasis library        (taken into accound only if key_oasis3 is activated)
+#
+# FC          Fortran compiler command
+# FCFLAGS     Fortran compiler flags
+# FFLAGS      Fortran 77 compiler flags
+# LD          linker
+# LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries
+# FPPFLAGS    pre-processing flags
+# AR          assembler
+# ARFLAGS     assembler flags
+# MK          make
+# USER_INC    complete list of include files
+# USER_LIB    complete list of libraries to pass to the linker
+# CC          C compiler used to compile conv for AGRIF
+# CFLAGS      compiler flags used with CC
+#
+# Note that:
+#  - unix variables "$..." are accpeted and will be evaluated before calling fcm.
+#  - fcm variables are starting with a % (and not a $)
+#
+
+%NCDF_HOME           /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/netcdf/4.6.1-intel-rdopmwr/
+%NCDF_HOME2          /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/netcdf-fortran/4.4.4-intel-mq54rwz/
+%HDF5_HOME           /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/hdf5/1.10.2-intel-726msh6/
+%CURL                /dss/dsshome1/lrz/sys/spack/release/19.2/opt/x86_avx512/curl/7.60.0-gcc-u7vewcb/lib/																							   
+%XIOS_HOME           /hppfs/work/pn68so/di67wat/NEMO/xios-2.5/
+%OASIS_HOME          /not/defined
+
+
+%HDF5_LIB            -L%HDF5_HOME/lib -L%CURL -lhdf5_hl -lhdf5
+%GCCLIB              
+
+
+%NCDF_INC            -I%NCDF_HOME/include -I%NCDF_HOME2/include -I%HDF5_HOME/include
+%NCDF_LIB            -L%NCDF_HOME/lib %HDF5_LIB -L%CURL -L%NCDF_HOME2/lib -lnetcdff -lnetcdf -L%GCCLIB -lstdc++ -lz -lcurl -lgpfs
+%XIOS_INC            -I%XIOS_HOME/inc
+%XIOS_LIB            -L%XIOS_HOME/lib -lxios -L%GCCLIB -lstdc++
+
+%OASIS_INC           -I%OASIS_HOME/build/lib/mct -I%OASIS_HOME/build/lib/psmile.MPI1
+%OASIS_LIB           -L%OASIS_HOME/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip
+
+
+%CPP                 icc -E
+%FC                  mpiifort
+%FCFLAGS             -O3 -r8 -funroll-all-loops -traceback
+
+
+
+%FFLAGS              %FCFLAGS
+%LD                  mpiifort
+%LDFLAGS			-lstdc++ -lifcore -O3 -traceback
+
+
+%FPPFLAGS            -P -C -traditional 
+
+%AR                  ar
+%ARFLAGS             -r
+
+%MK                  make
+%USER_INC            %XIOS_INC %OASIS_INC %NCDF_INC 
+%USER_LIB            %XIOS_LIB %OASIS_LIB %NCDF_LIB 
+
+
+
+%CC                  cc
+%CFLAGS              -O0
+

nemo/architecture_files/XIOS/arch-JUWELS.env

+module load GCC/8.3.0
+module load PGI/19.10-GCC-8.3.0
+module load Intel/2019.5.281-GCC-8.3.0
+module load ParaStationMPI/5.4
+module load HDF5/1.10.5
+module load netCDF/4.6.3
+module load netCDF-Fortran/4.4.5
+module load cURL
+module load Perl

nemo/architecture_files/XIOS/arch-JUWELS.fcm

+################################################################################
+###################                Projet XIOS               ###################
+################################################################################
+
+%CCOMPILER      mpicc
+%FCOMPILER      mpif90
+%LINKER         mpif90  -nofor-main
+
+%BASE_CFLAGS    -ansi -w
+%PROD_CFLAGS    -O3 -DBOOST_DISABLE_ASSERTS
+%DEV_CFLAGS     -g -O2 
+%DEBUG_CFLAGS   -g 
+
+%BASE_FFLAGS    -D__NONE__  -ffree-line-length-none
+%PROD_FFLAGS    -O3
+%DEV_FFLAGS     -g -O2
+%DEBUG_FFLAGS   -g 
+
+%BASE_INC       -D__NONE__
+%BASE_LD        -lstdc++
+
+%CPP            cpp
+%FPP            cpp -P
+%MAKE           make

nemo/architecture_files/XIOS/arch-JUWELS.path

+NETCDF_INCDIR="-I$NETCDF_INC_DIR -I$NETCDFF_INC_DIR"
+NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L$NETCDF_LIB_DIR -L$NETCDFF_LIB_DIR"
+NETCDF_LIB="-lnetcdff -lnetcdf"
+
+MPI_INCDIR=""
+MPI_LIBDIR=""
+MPI_LIB=""
+
+HDF5_INCDIR="-I $HDF5_INC_DIR"
+HDF5_LIBDIR="-L $HDF5_LIB_DIR"
+HDF5_LIB="-lhdf5_hl -lhdf5 -lz -lcurl"
+
+BOOST_INCDIR="-I $BOOST_INC_DIR"
+BOOST_LIBDIR="-L $BOOST_LIB_DIR"
+BOOST_LIB=""
+
+OASIS_INCDIR="-I$PWD/../../oasis3-mct/BLD/build/lib/psmile.MPI1"
+OASIS_LIBDIR="-L$PWD/../../oasis3-mct/BLD/lib"
+OASIS_LIB="-lpsmile.MPI1 -lscrip -lmct -lmpeu"
+

nemo/architecture_files/XIOS/arch-M100.env

+module load gnu
+module load szip
+module load zlib
+module load spectrum_mpi
+module load hdf5
+module load netcdf
+module load netcdff

nemo/architecture_files/XIOS/arch-M100.fcm

+################################################################################
+###################                Projet XIOS               ###################
+################################################################################
+
+%CCOMPILER      mpicc
+%FCOMPILER      mpif90
+%LINKER         mpif90  
+
+%BASE_CFLAGS    -ansi -w 
+%PROD_CFLAGS    -O3 -DBOOST_DISABLE_ASSERTS
+%DEV_CFLAGS     -g -O2  -traceback
+%DEBUG_CFLAGS   -DBZ_DEBUG -g -traceback -fno-inline
+
+%BASE_FFLAGS    -D__NONE__ -ffree-line-length-none
+%PROD_FFLAGS    -O3
+%DEV_FFLAGS     -g -O2 -traceback
+%DEBUG_FFLAGS   -g -traceback
+
+%BASE_INC       -D __NONE__
+%BASE_LD        -lstdc++
+
+%CPP            cpp
+%FPP            cpp -P
+%MAKE           make

nemo/architecture_files/XIOS/arch-M100.path

+NETCDF_INCDIR="-I$NETCDF_INC_DIR -I$NETCDFF_INC_DIR"
+NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L$NETCDF_LIB_DIR -L$NETCDFF_LIB_DIR"
+NETCDF_LIB="-lnetcdff -lnetcdf"
+
+MPI_INCDIR=""
+MPI_LIBDIR=""
+MPI_LIB=""
+
+HDF5_INCDIR="-I $HDF5_INC_DIR"
+HDF5_LIBDIR="-L $HDF5_LIB_DIR"
+HDF5_LIB="-lhdf5_hl -lhdf5  -lz -lcurl"
+
+BOOST_INCDIR="-I $BOOST_INC_DIR"
+BOOST_LIBDIR="-L $BOOST_LIB_DIR"
+BOOST_LIB=""
+
+OASIS_INCDIR="-I$PWD/../../oasis3-mct/BLD/build/lib/psmile.MPI1"
+OASIS_LIBDIR="-L$PWD/../../oasis3-mct/BLD/lib"
+OASIS_LIB="-lpsmile.MPI1 -lscrip -lmct -lmpeu"

nemo/architecture_files/XIOS/arch-SuperMUC.env

+module load slurm_setup
+module load hdf5
+module load netcdf
+module load netcdf-fortran

nemo/architecture_files/XIOS/arch-SuperMUC.fcm

+################################################################################
+###################                Projet XIOS               ###################
+################################################################################
+
+%CCOMPILER      mpicc
+%FCOMPILER      mpif90
+%LINKER         mpif90 -nofor-main
+
+%BASE_CFLAGS    -ansi -w  
+%PROD_CFLAGS    -O3 -DBOOST_DISABLE_ASSERTS
+%DEV_CFLAGS     -g -O2 
+%DEBUG_CFLAGS   -g 
+
+%BASE_FFLAGS    -D__NONE__ -ffree-line-length-none 
+%PROD_FFLAGS    -O3
+%DEV_FFLAGS     -g -O2
+%DEBUG_FFLAGS   -g 
+
+%BASE_INC       -D__NONE__
+%BASE_LD        -lstdc++
+
+%CPP            cpp
+%FPP            cpp -P
+%MAKE           make

nemo/architecture_files/XIOS/arch-SuperMUC.path

+NETCDF_INCDIR="-I$NETCDF_INC_DIR -I$NETCDFF_INC_DIR"
+NETCDF_LIBDIR="-Wl,'--allow-multiple-definition' -L $NETCDF_LIB_DIR -L $NETCDFF_LIB_DIR"
+NETCDF_LIB="-lnetcdff -lnetcdf"
+
+MPI_INCDIR=""
+MPI_LIBDIR=""
+MPI_LIB=""
+
+HDF5_INCDIR="-I $HDF5_INC_DIR"
+HDF5_LIBDIR="-L $HDF5_LIB_DIR"
+HDF5_LIB="-lhdf5_hl -lhdf5 -lz -lcurl"
+
+BOOST_INCDIR="-I $BOOST_INC_DIR"
+BOOST_LIBDIR="-L $BOOST_LIB_DIR"
+BOOST_LIB=""
+
+OASIS_INCDIR="-I$PWD/../../oasis3-mct/BLD/build/lib/psmile.MPI1"
+OASIS_LIBDIR="-L$PWD/../../oasis3-mct/BLD/lib"
+OASIS_LIB="-lpsmile.MPI1 -lscrip -lmct -lmpeu"

nemo/fixfcm.bash

-#!/usr/bin/env bash
-
-# A tool to modify XML files used by FCM
-
-# This is just a regexp search and replace, not a proper XML
-# parser. Use at own risk.
-
-fixfcm() {
-    local name value prog=""
-    for arg in "$@"; do
-        name="${arg%%=*}"
-	value=$(printf %q "${arg#*=}")
-	value="${value//\//\/}"
-        prog="s/(^%${name} )(.*)/\\1 ${value}/"$'\n'"$prog"
-    done
-    sed -r -e "$prog"
-}