diff --git a/README.md b/README.md
index f01d24a2ca44087f8390ef1fea667da4a9f1164c..8d69d3fecf94937cae3d5e9640a05a68da9ee6d3 100644
--- a/README.md
+++ b/README.md
@@ -282,7 +282,7 @@ The SHOC benchmark suite currently contains benchmark programs, categoried based
# SPECFEM3D
| **General information** | **Scientific field** | **Language** | **MPI** | **OpenMP** | **GPU** | **LoC** | **Code description** |
|------------------|----------------------|--------------|---------|------------|---------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [- Website](https://geodynamics.org/cig/software/specfem3d_globe/)
[- Source](https://github.com/geodynamics/specfem3d_globe.git)
[- Bench](https://repository.prace-ri.eu/git/UEABS/ueabs/tree/r2.1-dev/specfem3d)
[- Summary](https://repository.prace-ri.eu/git/UEABS/ueabs/blob/r2.1-dev/specfem3d/PRACE_UEABS_Specfem3D_summary.pdf) | Geodynamics | Fortran | yes | yes | Yes (CUDA) | 140000 | The software package SPECFEM3D simulates three-dimensional global and regional seismic wave propagation based upon the spectral-element method (SEM). |
+| [- Website](https://geodynamics.org/cig/software/specfem3d_globe/)
[- Source](https://github.com/geodynamics/specfem3d_globe.git)
[- Bench](https://repository.prace-ri.eu/git/UEABS/ueabs/tree/r2.1-dev/specfem3d)
[- Summary](https://repository.prace-ri.eu/git/UEABS/ueabs/blob/r2.1-dev/specfem3d/PRACE_UEABS_Specfem3D_summary.pdf) | Geodynamics | Fortran & C | yes | yes | Yes (CUDA) | 100k Fortran & 20k C | The software package SPECFEM3D simulates three-dimensional global and regional seismic wave propagation based upon the spectral-element method (SEM). |
# TensorFlow
diff --git a/gpaw/README.md b/gpaw/README.md
index a1b643a19e1c8fbd5967a0552dbcf5485b02263b..f39b861188e2b2f20f8fc8be5dced89534c8de75 100644
--- a/gpaw/README.md
+++ b/gpaw/README.md
@@ -100,7 +100,7 @@ these versions of GPAW.
Installing and running GPAW has changed a lot in the since the previous
versions of the UEABS. GPAW version numbering changed in 2019. Version 1.5.3 is the
last version with the old numbering. In 2019 the development team switched
-to a version numbering scheme based on year, month and patchlevel, e.g.,
+to a version numbering scheme based on year, month and patch level, e.g.,
19.8.1 for the second version released in August 2019.
Another change is in the Python packages used to install GPAW. Versions up to
@@ -169,9 +169,9 @@ In addition, the GPU version needs:
Installing GPAW also requires a number of standard build tools on the system, including
* [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
- configure script for libxc
+ configure script for LibXC
* [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
- the configure process of libxc produces very misleading
+ the configure process of LibXC produces very misleading
error messages that do not immediately point to libtool missing.
* [GNU make](https://www.gnu.org/software/make/)
@@ -297,8 +297,8 @@ on one hand and version 21.1.0 on the other hand.
The expected values are:
* Number of iterations: Between 30 and 35
- * Dipole (3rd component):between -0.493 and -0.491
- * Fermi level:between -2.67 and -2.66
+ * Dipole (3rd component): Between -0.493 and -0.491
+ * Fermi level: Between -2.67 and -2.66
* Extrapolated energy: Between -3784 and -3783
Note: Though not used for the benchmarking in the final report, some testing was done
@@ -307,7 +307,7 @@ by new internal implementations that cause changes in some results. For 21.1.0,
expected values are:
* Number of iterations: Between 30 and 35
- * Dipole (3rd component): between -0.462 and -0.461
- * Fermi level: between -2.59 and -2.58
+ * Dipole (3rd component): Between -0.462 and -0.461
+ * Fermi level: Between -2.59 and -2.58
* Extrapolated energy: Between -3784 and -3783
diff --git a/gpaw/build/build-CPU.md b/gpaw/build/build-CPU.md
index efa31904589b52aeb8ba13892b8f46e7c1acf5a9..a71399ebad6028c77fd3b27475d31e74595cbd62 100644
--- a/gpaw/build/build-CPU.md
+++ b/gpaw/build/build-CPU.md
@@ -1,4 +1,4 @@
-# Detailed GPAW installation instructions on non-acclerated systems
+# Detailed GPAW installation instructions on non-accelerated systems
These instructions are in addition to the brief instructions in [README.md](../README.md).
@@ -13,7 +13,7 @@ GPAW needs (for the UEABS benchmarks)
* [Python](https://www.python.org/): GPAW 20.1.0 requires Python 3.5-3.8, and
GPAW 20.10.0 and 21.1.0 require Python 3.6-3.9.
* [MPI library](https://www.mpi-forum.org/)
- * [LibXC](https://www.tddft.org/programs/libxc/). GPAW 20.1.0,
+ * [LibXC](https://www.tddft.org/programs/LibXC/). GPAW 20.1.0,
20.10.0 and 21.1.0 all need LibXC 3.x or 4.x.
* (Optimized) [BLAS](http://www.netlib.org/blas/) and
[LAPACK](http://www.netlib.org/lapack/) libraries.
@@ -22,7 +22,7 @@ GPAW needs (for the UEABS benchmarks)
will give very poor performance. Most optimized LAPACK libraries actually only
optimize a few critical routines while the remaining routines are compiled from
the reference version. Most processor vendors for HPC machines and system vendors
- offer optmized versions of these libraries.
+ offer optimized versions of these libraries.
* [ScaLAPACK](http://www.netlib.org/scalapack/) and the underlying communication
layer [BLACS](http://www.netlib.org/blacs/).
* [FFTW](http://www.fftw.org/) or compatible FFT library.
@@ -60,7 +60,7 @@ GPAW needs
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW. The required versions is 3.18.0 or later for
GPAW 20.1.0, 20.10.0 and 21.1.0.
- ASE has a couple of dependendencies
+ ASE has a couple of dependencies
that are not needed for running the UEABS benchmarks. However, several Python
package install methods will trigger the installation of those packages, and
with them may require a chain of system libraries.
@@ -69,7 +69,7 @@ GPAW needs
This package is optional and not really needed to run the benchmarks.
Matplotlib pulls in a lot of other dependencies. When installing ASE with pip,
it will try to pull in matplotlib and its dependencies
- * [pillow](https://pypi.org/project/Pillow/) needs several exgternal
+ * [pillow](https://pypi.org/project/Pillow/) needs several external
libraries. During the development of the benchmarks, we needed at least
zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even
though the pillow documentation claimed that libjpeg was optional,
@@ -90,7 +90,7 @@ GPAW needs
code
* [itsdangerous](https://pypi.org/project/itsdangerous/)
* [Werkzeug](https://pypi.org/project/Werkzeug/)
- * [click]()
+ * [click](https://pypi.org/project/click/)
## Tested configurations
@@ -145,28 +145,28 @@ Also, the instructions below will need to be adapted to the specific
libraries that are being used.
Other prerequisites:
- * libxc
+ * LibXC
* Python interpreter
* Python package NumPy
* Python package SciPy
* Python package ase
-### Installing libxc
+### Installing LibXC
- * Installing libxc requires GNU automake and GNU buildtool besides GNU make and a
+ * Installing LibXC requires GNU automake and GNU buildtool besides GNU make and a
C compiler. The build process is the usual GNU configure - make - make install
cycle, but the `configure` script still needs to be generated with autoreconf.
- * Download libxc:
- * The latest version of libxc can be downloaded from
- [the libxc download page](https://www.tddft.org/programs/libxc/download/).
+ * Download LibXC:
+ * The latest version of LibXC can be downloaded from
+ [the LibXC download page](https://www.tddft.org/programs/libxc/download/).
However, that version may not be officially supported by GPAW.
- * It is also possible to download all recent versions of libxc from
- [the libxc GitLab](https://gitlab.com/libxc/libxc)
+ * It is also possible to download all recent versions of LibXC from
+ [the LibXC GitLab](https://gitlab.com/libxc/libxc)
* Select the tag corresponding to the version you want to download in the
branch/tag selection box.
* Then use the download button and select the desired file type.
- * Dowload URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
+ * Download URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
* Untar the file in the build directory.
@@ -228,8 +228,8 @@ of NumPy, SciPy and GPAW itself proves much more important.
use the NumPy FFT routines.
* GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files
can be found on the [GPAW website, Atomic PAW Setups page](https://wiki.fysik.dtu.dk/gpaw/setups/setups.html).
- For the testing we used []`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
- for all versions of GPAW. The easiest way to install these files is to simpy untar
+ For the testing we used [`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
+ for all versions of GPAW. The easiest way to install these files is to simply untar
the file and set the environment variable GPAW_SETUP_PATH to point to that directory.
In the examples provided we use the `share/gpaw-setups` subdirectory of the install
directory for this purpose.
diff --git a/gpaw/build/examples/BSC-MareNostrum4-skylake/build_20.1.0_Python38_FFTW_icc.BSC.sh b/gpaw/build/examples/BSC-MareNostrum4-skylake/build_20.1.0_Python38_FFTW_icc.BSC.sh
new file mode 100755
index 0000000000000000000000000000000000000000..4d42553486fad8860d6b34c077092af0ce522616
--- /dev/null
+++ b/gpaw/build/examples/BSC-MareNostrum4-skylake/build_20.1.0_Python38_FFTW_icc.BSC.sh
@@ -0,0 +1,1072 @@
+#!/bin/bash
+#
+# Installation script for GPAW 20.1.0:
+# * We compile our own Python as this is the best guarantee to not have to
+# struggle with compatibility problems between various compilers used for
+# various components
+# * Using the matching version of ase, 3.19.3
+# * Compiling with the Intel compilers
+#
+# The FFT library is discovered at runtime. With the settings used in this script
+# this should be MKL FFT, but it is possible to change this at runtime to either
+# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
+# (link below).
+#
+# The original installation instructions for GPAW can be found at
+# https://gitlab.com/gpaw/gpaw/-/blob/20.1.0/doc/install.rst
+#
+
+packageID='20.1.0-Python38-FFTW-icc'
+packageName='GPAW-UEABS'
+
+echo -e "\n###### Building $packageName/$packageID from $0\n\n"
+
+# The next three variables are only used to load the right UEABS module
+# and to give example values for variable settings in comments.
+install_root=$PROJECT/UEABS
+systemID=BSC-MareNostrum4-skylake
+
+module purge
+MODULEPATH=$install_root/$systemID/Modules:$MODULEPATH
+
+#
+# The following UEABS_ variables are needed:
+# We set them manually as we have no UEABS module for this system at the moment.
+#
+# Directory to put the downloaded sources of the packages.
+UEABS_DOWNLOADS=$install_root/SOURCES
+# Directory where packages should be installed.
+UEABS_PACKAGES=$install_root/$systemID/Packages
+# Directory where modules are installed
+UEABS_MODULES=$install_root/$systemID/Modules
+
+install_dir=$UEABS_PACKAGES/$packageName/$packageID
+modules_dir=$UEABS_MODULES/$packageName
+#build_dir="/dev/shm/$USER/$packageName/$packageID"
+build_dir="$SCRATCH/UEABS-tmp/$packageName/$packageID"
+
+# Software versions
+python_version='3.8.7'
+zlib_version='1.2.11'
+ncurses_version='6.2'
+readline_version='8.0'
+sqlite_version='3.33.0'
+sqlite_download='3330000'
+libffi_version='3.3'
+fftw_version='3.3.8'
+libxc_version='4.3.4'
+
+setuptools_version='56.0.0'
+setuptoolsscm_version='6.0.1'
+wheel_version='0.35.1'
+
+attrs_version='20.3.0'
+pybind11_version='2.6.2'
+cython_version='0.29.21'
+
+py_version='1.10.0'
+pyparsing_version='2.4.7'
+toml_version='0.10.2'
+iniconfig_version='1.1.1'
+packaging_version='20.9'
+pytest_version='6.2.3'
+
+numpy_version='1.18.5'
+scipy_version='1.5.4'
+ase_version='3.19.3'
+GPAW_version='20.1.0'
+
+GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
+
+# Compiler settings
+#compiler_module='intel/2020.1'
+#mpi_module='impi/2018.4'
+#math_module='mkl/2020.1'
+compiler_module='intel/2018.4'
+mpi_module='impi/2018.4'
+math_module='mkl/2018.4'
+opt_level='-O2'
+proc_opt_flags='-xHost'
+fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
+parallel=16
+
+py_maj_min='3.8'
+
+################################################################################
+#
+# Prepare the system
+#
+
+#
+# Load modules
+#
+mkdir -p $modules_dir
+module load $compiler_module
+module load $mpi_module
+module load $math_module
+
+#
+# Create the directories and make sure they are clean if that matters
+#
+/usr/bin/mkdir -p $UEABS_DOWNLOADS
+
+/usr/bin/mkdir -p $install_dir
+/usr/bin/rm -rf $install_dir
+/usr/bin/mkdir -p $install_dir
+
+/usr/bin/mkdir -p $modules_dir
+
+/usr/bin/mkdir -p $build_dir
+/usr/bin/rm -rf $build_dir
+/usr/bin/mkdir -p $build_dir
+
+
+################################################################################
+#
+# Download components
+#
+
+echo -e "\n### Downloading files...\n"
+
+function wget() {
+
+ echo "Please download $1 to $UEABS_DOWNLOADS"
+
+}
+
+cd $UEABS_DOWNLOADS
+
+downloads_OK=1
+
+# zlib: https://www.zlib.net/zlib-1.2.11.tar.gz
+zlib_file="zlib-$zlib_version.tar.gz"
+zlib_url="https://www.zlib.net"
+[[ -f $zlib_file ]] || wget "$zlib_url/$zlib_file"
+[[ -f $zlib_file ]] || downloads_OK=0
+
+# ncurses: https://ftp.gnu.org/pub/gnu/ncurses/ncurses-6.2.tar.gz
+ncurses_file="ncurses-$ncurses_version.tar.gz"
+ncurses_url="https://ftp.gnu.org/pub/gnu/ncurses"
+[[ -f $ncurses_file ]] || wget "$ncurses_url/$ncurses_file"
+[[ -f $ncurses_file ]] || downloads_OK=0
+
+# readline: https://ftp.gnu.org/pub/gnu/readline/readline-8.0.tar.gz
+readline_file="readline-$readline_version.tar.gz"
+readline_url="https://ftp.gnu.org/pub/gnu/readline"
+[[ -f $readline_file ]] || wget "$readline_url/$readline_file"
+[[ -f $readline_file ]] || downloads_OK=0
+
+# sqlite: https://www.sqlite.org/2020/sqlite-autoconf-3330000.tar.gz
+sqlite_file="sqlite-autoconf-$sqlite_download.tar.gz"
+sqlite_url="https://www.sqlite.org/2020"
+[[ -f $sqlite_file ]] || wget "$sqlite_url/$sqlite_file"
+[[ -f $sqlite_file ]] || downloads_OK=0
+
+# libffi: https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz
+libffi_file="libffi-$libffi_version.tar.gz"
+libffi_url="https://github.com/libffi/libffi/releases/download/v$libffi_version"
+[[ -f $libffi_file ]] || wget "$libffi_url/$libffi_file"
+[[ -f $libffi_file ]] || downloads_OK=0
+
+# FFTW: http://www.fftw.org/fftw-3.3.8.tar.gz
+fftw_file="fftw-$fftw_version.tar.gz"
+fftw_url="http://www.fftw.org"
+[[ -f $fftw_file ]] || wget "$fftw_url/$fftw_file"
+[[ -f $fftw_file ]] || downloads_OK=0
+
+# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
+libxc_file="libxc-$libxc_version.tar.bz2"
+libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
+[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
+[[ -f $libxc_file ]] || downloads_OK=0
+
+# Python: https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz
+python_file="Python-$python_version.tar.xz"
+python_url="https://www.python.org/ftp/python/$python_version"
+[[ -f $python_file ]] || wget "$python_url/$python_file"
+[[ -f $python_file ]] || downloads_OK=0
+
+# Downloading setuptools.
+setuptools_file="setuptools-$setuptools_version.tar.gz"
+setuptools_url="https://pypi.python.org/packages/source/s/setuptools"
+[[ -f $setuptools_file ]] || wget $setuptools_url/$setuptools_file
+[[ -f $setuptools_file ]] || downloads_OK=0
+
+# Downloading setuptoolssscm so that we can gather all sources for reproducibility.
+setuptoolsscm_file="setuptools_scm-$setuptoolsscm_version.tar.gz"
+setuptoolsscm_url="https://pypi.python.org/packages/source/s/setuptools_scm"
+[[ -f $setuptoolsscm_file ]] || wget $setuptoolsscm_url/$setuptoolsscm_file
+[[ -f $setuptoolsscm_file ]] || downloads_OK=0
+
+# Downloading wheel so that we can gather all sources for reproducibility.
+wheel_file="wheel-$wheel_version.tar.gz"
+wheel_url="https://pypi.python.org/packages/source/w/wheel"
+[[ -f $wheel_file ]] || wget "$wheel_url/$wheel_file"
+[[ -f $wheel_file ]] || downloads_OK=0
+
+# Downloading attrs so that we can gather all sources for reproducibility.
+attrs_file="attrs-$attrs_version.tar.gz"
+attrs_url="https://pypi.python.org/packages/source/a/attrs"
+[[ -f $attrs_file ]] || wget $attrs_url/$attrs_file
+[[ -f $attrs_file ]] || downloads_OK=0
+
+# Downloading pybind11 so that we can gather all sources for reproducibility.
+pybind11_file="pybind11-$pybind11_version.tar.gz"
+pybind11_url="https://pypi.python.org/packages/source/p/pybind11"
+[[ -f $pybind11_file ]] || wget $pybind11_url/$pybind11_file
+[[ -f $pybind11_file ]] || downloads_OK=0
+
+# Downloading Cython so that we can gather all sources for reproducibility.
+cython_file="Cython-$cython_version.tar.gz"
+cython_url="https://pypi.python.org/packages/source/c/cython"
+[[ -f $cython_file ]] || wget "$cython_url/$cython_file"
+[[ -f $cython_file ]] || downloads_OK=0
+
+# Downloading py so that we can gather all sources for reproducibility.
+py_file="py-$py_version.tar.gz"
+py_url="https://pypi.python.org/packages/source/p/py"
+[[ -f $py_file ]] || wget $py_url/$py_file
+[[ -f $py_file ]] || downloads_OK=0
+
+# Downloading pyparsing so that we can gather all sources for reproducibility.
+pyparsing_file="pyparsing-$pyparsing_version.tar.gz"
+pyparsing_url="https://pypi.python.org/packages/source/p/pyparsing"
+[[ -f $pyparsing_file ]] || wget $pyparsing_url/$pyparsing_file
+[[ -f $pyparsing_file ]] || downloads_OK=0
+
+# Downloading toml so that we can gather all sources for reproducibility.
+toml_file="toml-$toml_version.tar.gz"
+toml_url="https://pypi.python.org/packages/source/t/toml"
+[[ -f $toml_file ]] || wget $toml_url/$toml_file
+[[ -f $toml_file ]] || downloads_OK=0
+
+# Downloading iniconfig so that we can gather all sources for reproducibility.
+iniconfig_file="iniconfig-$iniconfig_version.tar.gz"
+iniconfig_url="https://pypi.python.org/packages/source/i/iniconfig"
+[[ -f $iniconfig_file ]] || wget $iniconfig_url/$iniconfig_file
+[[ -f $iniconfig_file ]] || downloads_OK=0
+
+# Downloading packaging so that we can gather all sources for reproducibility.
+packaging_file="packaging-$packaging_version.tar.gz"
+packaging_url="https://pypi.python.org/packages/source/p/packaging"
+[[ -f $packaging_file ]] || wget $packaging_url/$packaging_file
+[[ -f $packaging_file ]] || downloads_OK=0
+
+# Downloading pytest so that we can gather all sources for reproducibility.
+pytest_file="pytest-$pytest_version.tar.gz"
+pytest_url="https://pypi.python.org/packages/source/p/pytest"
+[[ -f $pytest_file ]] || wget $pytest_url/$pytest_file
+[[ -f $pytest_file ]] || downloads_OK=0
+
+# NumPy needs customizations, so we need to download and unpack the sources
+numpy_file="numpy-$numpy_version.zip"
+numpy_url="https://pypi.python.org/packages/source/n/numpy"
+[[ -f $numpy_file ]] || wget "$numpy_url/$numpy_file"
+[[ -f $numpy_file ]] || downloads_OK=0
+
+# SciPy
+scipy_file="scipy-$scipy_version.tar.gz"
+scipy_url="https://pypi.python.org/packages/source/s/scipy"
+[[ -f $scipy_file ]] || wget "$scipy_url/$scipy_file"
+[[ -f $scipy_file ]] || downloads_OK=0
+
+# Downloading ase so that we can gather all sources for reproducibility
+ase_file="ase-$ase_version.tar.gz"
+ase_url="https://pypi.python.org/packages/source/a/ase"
+[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
+[[ -f $ase_file ]] || downloads_OK=0
+
+# GPAW needs customization, so we need to download and unpack the sources.
+GPAW_file="gpaw-$GPAW_version.tar.gz"
+GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
+[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
+[[ -f $GPAW_file ]] || downloads_OK=0
+
+# Download GPAW-setup, a number of setup files for GPAW.
+# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
+GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
+GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
+[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
+[[ -f $GPAWsetups_file ]] || downloads_OK=0
+
+[[ $downloads_OK ]] || exit
+
+
+################################################################################
+#
+# Set PATH-style variables
+#
+/usr/bin/mkdir -p $install_dir/bin
+PATH="$install_dir/bin:$PATH"
+/usr/bin/mkdir -p $install_dir/lib
+LD_LIBRARY_PATH="$install_dir/lib:$LD_LIBRARY_PATH"
+export LIBRARY_PATH="$LD_LIBRARY_PATH"
+
+
+################################################################################
+#
+# Install ncurses
+#
+# We mirror the two-step EasyBuild install process. This may be overkill, but
+# we know it works.
+#
+
+echo -e "\n### Installing ncurses...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$ncurses_file
+
+cd ncurses-$ncurses_version
+
+# Configure step 1
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export CXX=icpc
+export CXXFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --with-shared --enable-overwrite --without-ada --enable-symlinks
+
+# Build step 1
+make -j $parallel
+
+# Install step 1
+make install
+
+# Add bin, lib and include to the PATH variables
+PATH=$install_dir/bin:$PATH
+LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
+LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
+CPATH=$install_dir/include:$CPATH
+
+# Configure step 2
+make distclean
+./configure --prefix="$install_dir" \
+ --with-shared --enable-overwrite --without-ada --enable-symlinks \
+ --enable-ext-colors --enable-widec \
+ --includedir=$install_dir/include/ncursesw/
+
+# Build step 2
+make -j $parallel
+
+# Install step 2
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset CXXFLAGS
+echo -e "\n### Finishing ncurses installation...\n"
+
+
+################################################################################
+#
+# Install readline
+#
+
+echo -e "\n### Installing readline...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$readline_file
+
+cd readline-$readline_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export CPPFLAGS="-I$install_dir/include"
+export LDFLAGS="-L$install_dir/lib -lncurses"
+./configure --prefix="$install_dir"
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CPPFLAGS
+unset LDFLAGS
+echo -e "\n### Finishing readline installation...\n"
+
+
+################################################################################
+#
+# Install zlib
+#
+
+echo -e "\n### Installing zlib...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$zlib_file
+
+cd zlib-$zlib_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir"
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+echo -e "\n### Finishing zlib installation...\n"
+
+
+################################################################################
+#
+# Install libffi
+#
+
+echo -e "\n### Installing libffi...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libffi_file
+
+cd libffi-$libffi_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-multi-os-directory
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+echo -e "\n### Finishing libffi installation...\n"
+
+
+################################################################################
+#
+# Install SQLite
+#
+
+echo -e "\n### Installing SQLite...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$sqlite_file
+
+cd sqlite-autoconf-$sqlite_download
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC -DSQLITE_DISABLE_INTRINSIC"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir"
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CPPFLAGS
+echo -e "\n### Finishing SQLite installation...\n"
+
+
+################################################################################
+#
+# Install FFTW
+#
+
+echo -e "\n### Installing FFTW...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$fftw_file
+
+# Patch the sources to compile with icc
+cat >fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpiicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+export PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install wheel
+#
+# The BSC system does seem to need setuptools_sscm to install pytest.
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpiicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+export PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install wheel
+#
+# The BSC system does seem to need setuptools_sscm to install pytest.
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+#pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pluggy_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+#pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pluggy_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Install wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+#pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pluggy_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Install wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+#pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pluggy_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpiicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$modules_dir/$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpiicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j $parallel
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$download_dir $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$download_dir $UEABS_DOWNLOADS/$pybind11_file
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We don't care about version numbers here as it is not important for the
+# reproducibility of the benchmarks.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$modules_dir/$packageID.lua <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpiicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j 32
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\nInitialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PATH="$install_dir/bin:$PATH"
+LD_LIBRARY_PATH="$install_dir/lib:$LD_LIBRARY_PATH"
+LIBRARY_PATH="$install_dir/lib:$LIBRARY_PATH"
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install setuptools and wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+# setuptools_scm seems to be needed to install py.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We do need to care about version numbers because the LRZ system doesn't allow
+# downloading from the compute nodes, so we have to do everything manually in
+# advance.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$modules_dir/$packageID <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpiicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j 32
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\nInitialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+PATH="$install_dir/bin:$PATH"
+LD_LIBRARY_PATH="$install_dir/lib:$LD_LIBRARY_PATH"
+LIBRARY_PATH="$install_dir/lib:$LIBRARY_PATH"
+PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install setuptools and wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+# setuptools_scm seems to be needed to install py.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We do need to care about version numbers because the LRZ system doesn't allow
+# downloading from the compute nodes, so we have to do everything manually in
+# advance.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$modules_dir/$packageID <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j 32
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+export PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install setuptools and wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+# setuptools_scm seems to be needed to install py.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We do need to care about version numbers because the LRZ system doesn't allow
+# downloading from the compute nodes, so we have to do everything manually in
+# advance.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+#pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pluggy_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$modules_dir/$packageID <fftw.patch <&5
+ $as_echo_n "checking whether C compiler accepts -no-gcc... " >&6; }
+
+EOF
+
+cd fftw-$fftw_version
+patch -p1 <../fftw.patch
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+export MPICC=mpicc
+export F77=ifort
+export F77FLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-sse --disable-sse2 --disable-avx --enable-avx2 --disable-avx512 --disable-avx-128-fma --disable-kcvi \
+ --enable-fma \
+ --disable-openmp --disable-threads \
+ --enable-mpi \
+ -disable-static --enable-shared --disable-fortran
+
+# Build
+make -j $parallel
+
+# Install
+make install
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset MPICC
+unset F77
+unset F77FLAGS
+
+echo -e "\n### Finishing FFTW installation...\n"
+
+
+################################################################################
+#
+# Install libxc
+#
+
+echo -e "\n### Installing libxc...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$libxc_file
+
+cd libxc-$libxc_version
+
+# Configure
+autoreconf -i
+
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
+./configure --prefix="$install_dir" \
+ --disable-static --enable-shared --disable-fortran
+# Build
+make -j $parallel
+
+# Install
+make -j install
+
+# Clean-up
+unset CC
+unset CFLAGS
+
+echo -e "\n### Finishing libxc installation...\n"
+
+
+################################################################################
+#
+# Install Python
+#
+
+echo -e "\n### Installing Python...\n"
+
+cd $build_dir
+
+# Uncompress
+tar -xf $UEABS_DOWNLOADS/$python_file
+
+cd Python-$python_version
+
+# Configure
+export CC=icc
+export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC -diag-disable=1678,10148,111,169,188,3438,2650,3175,1890"
+export CXX=icpc
+export FC=ifort
+export F90=$FC
+export FFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fwrapv -fPIC"
+export LD=icc
+export LDFLAGS="-L$install_dir/lib"
+export CPPFLAGS="-I$install_dir/include"
+./configure --prefix="$install_dir" \
+ --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
+ --enable-shared --disable-ipv6 \
+ --enable-optimizations \
+ --with-ensurepip=upgrade
+# --with-icc \
+
+# Build
+make -j 32
+
+# Install
+make install
+cd $install_dir/bin
+ln -s python$py_maj_min python
+
+# Clean-up
+unset CC
+unset CFLAGS
+unset CXX
+unset FC
+unset F90
+unset FFLAGS
+unset LD
+unset LDFLAGS
+unset CPPFLAGS
+
+echo -e "\n### Finishing Python installation...\n"
+
+
+################################################################################
+#
+# Initialising for installing Python packages
+#
+
+echo -e "\n### Initialising for installing Python packages...\n"
+
+/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
+cd $install_dir
+/usr/bin/ln -s lib lib64
+export PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
+
+
+################################################################################
+#
+# Install setuptools and wheel
+#
+
+echo -e "\n### Installing wheel...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptools_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$wheel_file
+# setuptools_scm seems to be needed to install py.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$setuptoolsscm_file
+
+echo -e "\n### Finishing wheel installation...\n"
+
+
+################################################################################
+#
+# Some other Python packages that are needed
+#
+
+echo -e "\n### Installing additional Python packages...\n"
+
+cd $build_dir
+
+# attrs is needed for Cython and the optional pytest.
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$attrs_file
+# pybind11 is needed for scipy
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pybind11_file
+
+echo -e "\n### Finishing additional Python packages installation...\n"
+
+
+################################################################################
+#
+# Optional: Install pytest and its dependencies to test NumPy and SciPy with
+# import numpy
+# numpy.test()
+# import scipy
+# scipy.text()
+# We do need to care about version numbers because the LRZ system doesn't allow
+# downloading from the compute nodes, so we have to do everything manually in
+# advance.
+#
+
+echo -e "\n### Installing pytest...\n"
+
+cd $build_dir
+
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$toml_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pyparsing_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$py_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$iniconfig_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$packaging_file
+#pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pluggy_file
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$pytest_file
+
+echo -e "\n### Finishing pytest installation...\n"
+
+
+################################################################################
+#
+# Install Cython
+#
+
+echo -e "\n### Installing Cython...\n"
+
+cd $build_dir
+pip$py_maj_min install --prefix="$install_dir" --no-deps --ignore-installed --find-links=$UEABS_DOWNLOADS $UEABS_DOWNLOADS/$cython_file
+
+echo -e "\n### Finishing Cython installation...\n"
+
+
+################################################################################
+#
+# Install NumPy
+#
+
+echo -e "\n### Installing NumPy...\n"
+
+cd $build_dir
+
+# Uncompress
+unzip $UEABS_DOWNLOADS/$numpy_file
+
+cd numpy-$numpy_version
+
+cat >site.cfg <siteconfig.py <$modules_dir/$packageID <&1 | tee loki-inst
-cd ..
-sed -e "s||$tgt|g" -e "s||$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/davide/gpaw-cpu/setup/customize-power8.py b/gpaw/build/examples/davide/gpaw-cpu/setup/customize-power8.py
deleted file mode 100644
index a098bfb54a08be9c3c8474feb72ecf98f83ecb24..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw-cpu/setup/customize-power8.py
+++ /dev/null
@@ -1,39 +0,0 @@
-# Setup customisation for gpaw/cuda
-import os
-
-# compiler and linker
-compiler = 'gcc'
-mpicompiler = 'mpicc'
-mpilinker = 'mpicc'
-extra_compile_args = ['-std=c99', '-mcpu=power8']
-
-# libraries
-libraries = ['z']
-
-# openblas
-library_dirs += [os.environ['OPENBLAS_ROOT'] + '/lib']
-include_dirs += [os.environ['OPENBLAS_ROOT'] + '/include']
-libraries += ['openblas']
-
-# scalapack
-library_dirs += [os.environ['SCALAPACK_ROOT'] + '/lib']
-libraries += ['scalapack']
-
-# libxc
-library_dirs += [os.environ['LIBXCDIR'] + '/lib']
-include_dirs += [os.environ['LIBXCDIR'] + '/include']
-libraries += ['xc']
-
-# GPAW defines
-define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
-define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
-define_macros += [("GPAW_ASYNC",1)]
-define_macros += [("GPAW_MPI2",1)]
-define_macros += [('GPAW_CUDA', '1')]
-
-# ScaLAPACK
-scalapack = True
-
-# HDF5
-hdf5 = False
-
diff --git a/gpaw/build/examples/davide/gpaw-cpu/setup/load-gpaw.sh b/gpaw/build/examples/davide/gpaw-cpu/setup/load-gpaw.sh
deleted file mode 100644
index f7f01edf5454c2b9854ec37ecd9ee32330b06bb0..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw-cpu/setup/load-gpaw.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/bash
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-source /load.sh
-source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
-export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
-export PATH=/bin:$PATH
-export PYTHONPATH=/lib/python2.7/site-packages:$PYTHONPATH
diff --git a/gpaw/build/examples/davide/gpaw/build.sh b/gpaw/build/examples/davide/gpaw/build.sh
deleted file mode 100644
index 56ff5e88e4a6c25a2b5a54d32add958f764b5540..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw/build.sh
+++ /dev/null
@@ -1,36 +0,0 @@
-### GPAW installation script for D.A.V.I.D.E
-
-# version numbers (modify if needed)
-gpaw_version=cuda
-
-# installation directory (modify!)
-tgt=$CINECA_SCRATCH/lib/gpaw-${gpaw_version}
-
-# setup build environment
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-source $CINECA_SCRATCH/lib/python-2018-12-cuda/load.sh 2016-06
-source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
-export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
-export CFLAGS=""
-
-# gpaw
-git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
-cd gpaw-$gpaw_version
-git checkout $gpaw_version
-patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
-cp ../setup/customize-cuda.py .
-cd c/cuda
-cp ../../../setup/make.inc .
-make 2>&1 | tee loki-make
-cd -
-python setup.py install --customize=customize-cuda.py --prefix=$tgt 2>&1 | tee loki-inst
-cd ..
-sed -e "s||$tgt|g" -e "s||$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/davide/gpaw/setup/customize-cuda.py b/gpaw/build/examples/davide/gpaw/setup/customize-cuda.py
deleted file mode 100644
index d2aae9409832d719e06a66450032d8588e769149..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw/setup/customize-cuda.py
+++ /dev/null
@@ -1,44 +0,0 @@
-# Setup customisation for gpaw/cuda
-import os
-
-# compiler and linker
-compiler = 'gcc'
-mpicompiler = 'mpicc'
-mpilinker = 'mpicc'
-extra_compile_args = ['-std=c99', '-mcpu=power8']
-
-# libraries
-libraries = ['z']
-
-# cuda
-library_dirs += [os.environ['CUDA_LIB'], './c/cuda']
-include_dirs += [os.environ['CUDA_HOME'] + '/include']
-libraries += ['gpaw-cuda', 'cublas', 'cudart', 'stdc++']
-
-# openblas
-library_dirs += [os.environ['OPENBLAS_ROOT'] + '/lib']
-include_dirs += [os.environ['OPENBLAS_ROOT'] + '/include']
-libraries += ['openblas']
-
-# scalapack
-library_dirs += [os.environ['SCALAPACK_ROOT'] + '/lib']
-libraries += ['scalapack']
-
-# libxc
-library_dirs += [os.environ['LIBXCDIR'] + '/lib']
-include_dirs += [os.environ['LIBXCDIR'] + '/include']
-libraries += ['xc']
-
-# GPAW defines
-define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
-define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
-define_macros += [("GPAW_ASYNC",1)]
-define_macros += [("GPAW_MPI2",1)]
-define_macros += [('GPAW_CUDA', '1')]
-
-# ScaLAPACK
-scalapack = True
-
-# HDF5
-hdf5 = False
-
diff --git a/gpaw/build/examples/davide/gpaw/setup/load-gpaw.sh b/gpaw/build/examples/davide/gpaw/setup/load-gpaw.sh
deleted file mode 100644
index f7f01edf5454c2b9854ec37ecd9ee32330b06bb0..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw/setup/load-gpaw.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/bash
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-source /load.sh
-source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
-export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
-export PATH=/bin:$PATH
-export PYTHONPATH=/lib/python2.7/site-packages:$PYTHONPATH
diff --git a/gpaw/build/examples/davide/gpaw/setup/make.inc b/gpaw/build/examples/davide/gpaw/setup/make.inc
deleted file mode 100644
index 2fa2d24c2ffc12959844333af0e3df70a07b0f71..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw/setup/make.inc
+++ /dev/null
@@ -1,57 +0,0 @@
-####################################################################
-# make include file. #
-####################################################################
-#
-SHELL = /bin/sh
-
-# ----------------------------------------------------------------------
-# - gpaw-cuda Directory Structure / gpaw-cuda library --------------------
-# ----------------------------------------------------------------------
-#
-TOPdir = .
-INCdir = $(TOPdir)
-PYTHONINCdir ?= $(PYTHONHOME)/include/python2.7/
-PYTHONLIBdir ?= $(PYTHONHOME)/lib/
-NUMPYINCdir ?= `python -c "import numpy; print numpy.get_include()"`
-MPIINCdir ?= $(OPENMPI_HOME)/include
-LIBdir = $(TOPdir)
-CUGPAWLIB = $(LIBdir)/libgpaw-cuda.a
-
-#
-# ----------------------------------------------------------------------
-# - NVIDIA CUDA includes / libraries / specifics -----------------------
-# ----------------------------------------------------------------------
-CUDAINCdir = $(CUDADIR)/include
-CUDALIBdir = $(CUDADIR)/lib64
-CUDA_OPTS =
-
-#
-# ----------------------------------------------------------------------
-# - gpaw-cuda includes / libraries / specifics -------------------------------
-# ----------------------------------------------------------------------
-#
-
-CUGPAW_INCLUDES = -I$(INCdir) -I$(CUDAINCdir) -I$(MPIINCdir) -I$(NUMPYINCdir) -I$(PYTHONINCdir)
-CUGPAW_OPTS = -DPARALLEL=1 -DGPAW_CUDA=1
-
-#
-# ----------------------------------------------------------------------
-#
-
-CUGPAW_DEFS = $(CUGPAW_OPTS) $(CUDA_OPTS) $(CUGPAW_INCLUDES)
-
-#
-# ----------------------------------------------------------------------
-# - Compilers / linkers - Optimization flags ---------------------------
-# ----------------------------------------------------------------------
-CC = gcc
-CCNOOPT = $(CUGPAW_DEFS)
-CCFLAGS = $(CUGPAW_DEFS) -g -fPIC -std=c99 -m64 -O3
-
-NVCC = nvcc
-NVCCFLAGS = $(CUGPAW_DEFS) -O3 -g -gencode arch=compute_60,code=sm_60 -m64 --compiler-options '-O3 -g -std=c99 -fPIC'
-
-ARCH = ar
-ARCHFLAGS= cr
-RANLIB = ranlib
-
diff --git a/gpaw/build/examples/davide/gpaw/setup/patch-rmmdiis.diff b/gpaw/build/examples/davide/gpaw/setup/patch-rmmdiis.diff
deleted file mode 100644
index adefa27c31dd72c87df2d3e35b8fcb56a7224dcf..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/gpaw/setup/patch-rmmdiis.diff
+++ /dev/null
@@ -1,34 +0,0 @@
-commit 761cba649d58e2d2f24c0a1e2fdad917b5929679
-Author: Martti Louhivuori
-Date: Thu May 18 10:56:07 2017 +0300
-
- Remove obsolete error calculation from DIIS step
-
-diff --git a/gpaw/eigensolvers/rmmdiis.py b/gpaw/eigensolvers/rmmdiis.py
-index 7d60553..d182713 100644
---- a/gpaw/eigensolvers/rmmdiis.py
-+++ b/gpaw/eigensolvers/rmmdiis.py
-@@ -299,23 +299,6 @@ class RMMDIIS(Eigensolver):
- P_axi, kpt.eps_n[n_x], R_xG, n_x,
- calculate_change=True)
- self.timer.stop('Calculate residuals')
-- self.timer.start('Calculate errors')
-- errors_new_x = np.zeros(B)
-- # errors_x[:] = 0.0
-- for n in range(n1, n2):
-- if kpt.f_n is None:
-- weight = kpt.weight
-- else:
-- weight = kpt.f_n[n]
-- if self.nbands_converge != 'occupied':
-- if wfs.bd.global_index(n) < self.nbands_converge:
-- weight = kpt.weight
-- else:
-- weight = 0.0
-- errors_new_x[n-n1] += weight * integrate(R_xG[n - n1],
-- R_xG[n - n1])
-- comm.sum(errors_x)
-- self.timer.stop('Calculate errors')
-
- self.timer.stop('DIIS step')
- # Final trial step
diff --git a/gpaw/build/examples/davide/openblas/build.sh b/gpaw/build/examples/davide/openblas/build.sh
deleted file mode 100644
index e108514a230e476ff391e327e9bbd5bac0f5902a..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/openblas/build.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-### OpenBLAS installation script for D.A.V.I.D.E
-
-# version numbers (modify if needed)
-openblas_version=0.3.4
-
-# installation directory (modify!)
-tgt=$CINECA_SCRATCH/lib/openblas-${openblas_version}-openmp
-
-# setup build environment
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-export CC=gcc
-export CFLAGS='-mcpu=power8 -O3'
-export CXX=g++
-export CXXFLAGS='-mcpu=power8 -O3'
-export FC=gfortran
-export FFLAGS='-mcpu=power8 -O3'
-
-# openblas
-git clone https://github.com/xianyi/OpenBLAS
-cd OpenBLAS
-git checkout v$openblas_version
-make TARGET=POWER8 USE_OPENMP=1 2>&1 | tee loki-make
-make install PREFIX=$tgt 2>&1 | tee loki-install
-sed -e "s||$tgt|g" ../setup/load-openblas.sh > $tgt/load.sh
-cd ..
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/davide/openblas/setup/load-openblas.sh b/gpaw/build/examples/davide/openblas/setup/load-openblas.sh
deleted file mode 100644
index eb99f2f9d7c089cd3a372a288516b7798b5ec32b..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/openblas/setup/load-openblas.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-export OPENBLAS_ROOT=
-export LD_LIBRARY_PATH=$OPENBLAS_ROOT/lib:$LD_LIBRARY_PATH
diff --git a/gpaw/build/examples/davide/python/build-extra.sh b/gpaw/build/examples/davide/python/build-extra.sh
deleted file mode 100644
index 1b17596db68b62ce328e48e067d4fc89d5d02a4b..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/python/build-extra.sh
+++ /dev/null
@@ -1,84 +0,0 @@
-### Python (extra) modules installation script for D.A.V.I.D.E
-### uses PYTHONUSERBASE to bundle all modules into a separate location
-### away from the base python installation
-
-# load Python
-source $PYTHONHOME/load.sh
-
-# bundle ID (e.g. time of release) (modify if needed)
-bundle=2016-06
-
-# version numbers (modify if needed)
-numpy_version=1.10.4
-scipy_version=0.17.1
-ase_version=3.11.0
-pycuda_version=2017.1.1
-libxc_version=2.1.3
-
-# installation directory (modify!)
-tgt=$PYTHONHOME/bundle/$bundle
-
-# setup build environment
-export CFLAGS="-fPIC $CFLAGS"
-export FFLAGS="-fPIC $FFLAGS"
-
-# use --user to install modules
-export PYTHONUSERBASE=$tgt
-mkdir -p $PYTHONUSERBASE/lib/python2.7/site-packages
-
-# build in a separate directory
-mkdir bundle-$bundle
-cd bundle-$bundle
-
-# cython + mpi4py
-pip install --user cython
-pip install --user mpi4py
-
-# numpy
-git clone git://github.com/numpy/numpy.git numpy-$numpy_version
-cd numpy-$numpy_version
-git checkout v$numpy_version
-sed -e "s||$OPENBLAS_ROOT|g" ../../setup/davide-openblas.cfg > site.cfg
-python setup.py build -j 4 install --user 2>&1 | tee loki-inst
-cd ..
-
-# scipy
-git clone git://github.com/scipy/scipy.git scipy-$scipy_version
-cd scipy-$scipy_version
-git checkout v$scipy_version
-python setup.py build -j 4 install --user 2>&1 | tee loki-inst
-cd ..
-
-# ase
-git clone https://gitlab.com/ase/ase.git ase-$ase_version
-cd ase-$ase_version
-git checkout $ase_version
-python setup.py install --user 2>&1 | tee loki-inst
-cd ..
-
-# libxc
-tar xvfz ~/src/libxc-${libxc_version}.tar.gz
-cd libxc-$libxc_version
-./configure --prefix=$PYTHONUSERBASE --enable-shared | tee loki-conf
-make | tee loki-make
-make install | tee loki-inst
-export LD_LIBRARY_PATH=$PYTHONUSERBASE/lib:$LD_LIBRARY_PATH
-cd ..
-
-# pycuda
-pip install --user pycuda==$pycuda_version
-
-# go back to the main build directory
-cd ..
-
-# if this is the first bundle, use it as default
-if [ ! -e $PYTHONHOME/bundle/default ]
-then
- cd $PYTHONHOME/bundle
- ln -s $bundle default
- cd -
-fi
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/davide/python/build.sh b/gpaw/build/examples/davide/python/build.sh
deleted file mode 100644
index 6dbaeea61c402250233bbf8b62886f8718a3051d..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/python/build.sh
+++ /dev/null
@@ -1,40 +0,0 @@
-### Python installation script for D.A.V.I.D.E
-### uses --prefix to set a custom installation directory
-
-# version numbers (modify if needed)
-python_version=2.7.13
-
-# installation directory (modify!)
-tgt=$CINECA_SCRATCH/lib/python-2018-12-cuda
-
-# setup build environment
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-export CC=gcc
-export CFLAGS='-mcpu=power8 -O3'
-export CXX=g++
-export CXXFLAGS='-mcpu=power8 -O3'
-export F77=gfortran
-export FFLAGS='-mcpu=power8 -O3'
-
-# python
-git clone https://github.com/python/cpython.git python-$python_version
-cd python-$python_version
-git checkout v$python_version
-./configure --prefix=$tgt --enable-shared --disable-ipv6 --enable-unicode=ucs4 2>&1 | tee loki-conf
-make 2>&1 | tee loki-make
-make install 2>&1 | tee loki-inst
-cd ..
-sed -e "s||$tgt|g" setup/load-python.sh > $tgt/load.sh
-
-# install pip
-source $tgt/load.sh
-python -m ensurepip
-pip install --upgrade pip
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/davide/python/setup/davide-openblas.cfg b/gpaw/build/examples/davide/python/setup/davide-openblas.cfg
deleted file mode 100644
index ba027c31a76734ca1d69c30f38c0c4832d35c59a..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/python/setup/davide-openblas.cfg
+++ /dev/null
@@ -1,4 +0,0 @@
-[openblas]
-libraries = openblas
-library_dirs = /lib
-include_dirs = /include
diff --git a/gpaw/build/examples/davide/python/setup/load-python.sh b/gpaw/build/examples/davide/python/setup/load-python.sh
deleted file mode 100644
index 69b2033864e3bbf9f756946f78eb24f71cc4ddaa..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/python/setup/load-python.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/bin/bash
-export PYTHONHOME=
-export PYTHONPATH=$PYTHONHOME/lib
-export PATH=$PYTHONHOME/bin:$PATH
-export MANPATH=$PYTHONHOME/share/man:$MANPATH
-export LD_LIBRARY_PATH=$PYTHONHOME/lib:$LD_LIBRARY_PATH
-
-if [[ $# -gt 0 ]]
-then
- export PYTHONUSERBASE=$PYTHONHOME/bundle/$1
- export PATH=$PYTHONUSERBASE/bin:$PATH
- export LD_LIBRARY_PATH=$PYTHONUSERBASE/lib:$LD_LIBRARY_PATH
-elif [[ -e $PYTHONHOME/bundle/default ]]
-then
- export PYTHONUSERBASE=$PYTHONHOME/bundle/default
- export PATH=$PYTHONUSERBASE/bin:$PATH
- export LD_LIBRARY_PATH=$PYTHONUSERBASE/lib:$LD_LIBRARY_PATH
-fi
-if [[ -e $PYTHONUSERBASE/include/xc.h ]]
-then
- export LIBXCDIR=$PYTHONUSERBASE
-fi
diff --git a/gpaw/build/examples/davide/scalapack/build.sh b/gpaw/build/examples/davide/scalapack/build.sh
deleted file mode 100644
index 52a7bb31071ea15a099ee970a951042a5c2242c1..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/scalapack/build.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-### ScaLAPACK installation script for D.A.V.I.D.E
-
-# version numbers (modify if needed)
-scalapack_version=2.0.2
-
-# installation directory (modify!)
-tgt=$CINECA_SCRATCH/lib/scalapack-${scalapack_version}
-
-# setup build environment
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-export CFLAGS="-mcpu=power8 -O3"
-export FFLAGS="-mcpu=power8 -O3"
-
-# scalapack
-tar xvfz ~/scalapack-${scalapack_version}.tgz
-cd scalapack-${scalapack_version}
-cp ../setup/SLmake.inc .
-mkdir build
-cd build
-cmake -DBLAS_LIBRARIES=$OPENBLAS_ROOT/lib/libopenblas.so -DLAPACK_LIBRARIES=$OPENBLAS_ROOT/lib/libopenblas.so -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=$tgt ..
-make 2>&1 | tee loki-make
-make install 2>&1 | tee loki-install
-sed -e "s||$tgt|g" ../../setup/load-scalapack.sh > $tgt/load.sh
-cd ../..
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/davide/scalapack/setup/SLmake.inc b/gpaw/build/examples/davide/scalapack/setup/SLmake.inc
deleted file mode 100644
index 4d5f6aa38cc020b0ae04f9714a16febf21216b59..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/scalapack/setup/SLmake.inc
+++ /dev/null
@@ -1,61 +0,0 @@
-############################################################################
-#
-# Program: ScaLAPACK
-#
-# Module: SLmake.inc
-#
-# Purpose: Top-level Definitions
-#
-# Creation date: February 15, 2000
-#
-# Modified: October 13, 2011
-#
-# Send bug reports, comments or suggestions to scalapack@cs.utk.edu
-#
-############################################################################
-#
-# C preprocessor definitions: set CDEFS to one of the following:
-#
-# -DNoChange (fortran subprogram names are lower case without any suffix)
-# -DUpCase (fortran subprogram names are upper case without any suffix)
-# -DAdd_ (fortran subprogram names are lower case with "_" appended)
-
-CDEFS = -DAdd_
-
-#
-# The fortran and C compilers, loaders, and their flags
-#
-
-FC = mpif90
-CC = mpicc
-NOOPT = -mcpu=power8 -O0
-FCFLAGS = -mcpu=power8 -O3 -shared -fPIC
-CCFLAGS = -mcpu=power8 -O3 -shared -fPIC
-FCLOADER = $(FC)
-CCLOADER = $(CC)
-FCLOADFLAGS = $(FCFLAGS)
-CCLOADFLAGS = $(CCFLAGS)
-
-#
-# The archiver and the flag(s) to use when building archive (library)
-# Also the ranlib routine. If your system has no ranlib, set RANLIB = echo
-#
-
-ARCH = ar
-ARCHFLAGS = cr
-RANLIB = ranlib
-
-#
-# The name of the ScaLAPACK library to be created
-#
-
-SCALAPACKLIB = libscalapack.so
-
-#
-# BLAS, LAPACK (and possibly other) libraries needed for linking test programs
-#
-
-BLASLIB = -lopenblas -L$(OPENBLAS_ROOT)/lib -I$(OPENBLAS_ROOT)/include
-LAPACKLIB =
-LIBS = $(LAPACKLIB) $(BLASLIB)
-
diff --git a/gpaw/build/examples/davide/scalapack/setup/load-scalapack.sh b/gpaw/build/examples/davide/scalapack/setup/load-scalapack.sh
deleted file mode 100644
index c37538574e2d073453dba51efd66a5d7599649cb..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/davide/scalapack/setup/load-scalapack.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-export SCALAPACK_ROOT=
-export LD_LIBRARY_PATH=$SCALAPACK_ROOT/lib:$LD_LIBRARY_PATH
diff --git a/gpaw/build/examples/juwels/gpaw/build.sh b/gpaw/build/examples/juwels/gpaw/build.sh
deleted file mode 100644
index 5206b59856d7179d009f82c34a70e2d27a3a362c..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/gpaw/build.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-### GPAW installation script for JUWELS
-
-# version numbers (modify if needed)
-gpaw_version=1.1.0
-
-# installation directory (modify!)
-tgt=$HOME/lib/gpaw-${gpaw_version}
-
-# setup build environment
-module load CUDA/9.2.88
-module load Intel/2019.0.117-GCC-7.3.0
-module load IntelMPI/2018.4.274
-module load imkl/2019.0.117
-source $HOME/lib/python-2019-01/load.sh 2016-06
-export GPAW_SETUP_PATH=$HOME/lib/gpaw-setups-0.9.11271
-export CFLAGS=""
-
-# gpaw
-git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
-cd gpaw-$gpaw_version
-git checkout $gpaw_version
-patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
-cp ../setup/customize-juwels.py .
-python setup.py install --customize=customize-juwels.py --prefix=$tgt 2>&1 | tee loki-inst
-cd ..
-sed -e "s||$tgt|g" -e "s||$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/juwels/gpaw/setup/customize-juwels.py b/gpaw/build/examples/juwels/gpaw/setup/customize-juwels.py
deleted file mode 100644
index 7d7457df750f04cefd55523a6e1a6fb0cb2a646a..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/gpaw/setup/customize-juwels.py
+++ /dev/null
@@ -1,34 +0,0 @@
-# Setup customisation for gpaw/cuda
-import os
-
-# compiler and linker
-compiler = 'icc'
-mpicompiler = 'mpicc'
-mpilinker = 'mpicc'
-extra_compile_args = ['-std=c99']
-
-# libraries
-libraries = ['z']
-
-# use MKL
-library_dirs += [os.environ['MKLROOT'] + '/lib/intel64/']
-libraries = ['mkl_intel_lp64', 'mkl_sequential', 'mkl_core']
-mpi_libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
-
-# libxc
-library_dirs += [os.environ['LIBXCDIR'] + '/lib']
-include_dirs += [os.environ['LIBXCDIR'] + '/include']
-libraries += ['xc']
-
-# GPAW defines
-define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
-define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
-define_macros += [("GPAW_ASYNC",1)]
-define_macros += [("GPAW_MPI2",1)]
-
-# ScaLAPACK
-scalapack = True
-
-# HDF5
-hdf5 = False
-
diff --git a/gpaw/build/examples/juwels/gpaw/setup/load-gpaw.sh b/gpaw/build/examples/juwels/gpaw/setup/load-gpaw.sh
deleted file mode 100644
index 2bb4fe410fe5c1494b2994999ffd3c4f190aaf3d..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/gpaw/setup/load-gpaw.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-#!/bin/bash
-module load CUDA/9.2.88
-module load Intel/2019.0.117-GCC-7.3.0
-module load IntelMPI/2018.4.274
-module load imkl/2019.0.117
-source $HOME/lib/python-2019-01/load.sh 2016-06
-export GPAW_SETUP_PATH=$HOME/lib/gpaw-setups-0.9.11271
-export PATH=/bin:$PATH
-export PYTHONPATH=/lib/python2.7/site-packages:$PYTHONPATH
diff --git a/gpaw/build/examples/juwels/gpaw/setup/patch-rmmdiis.diff b/gpaw/build/examples/juwels/gpaw/setup/patch-rmmdiis.diff
deleted file mode 100644
index adefa27c31dd72c87df2d3e35b8fcb56a7224dcf..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/gpaw/setup/patch-rmmdiis.diff
+++ /dev/null
@@ -1,34 +0,0 @@
-commit 761cba649d58e2d2f24c0a1e2fdad917b5929679
-Author: Martti Louhivuori
-Date: Thu May 18 10:56:07 2017 +0300
-
- Remove obsolete error calculation from DIIS step
-
-diff --git a/gpaw/eigensolvers/rmmdiis.py b/gpaw/eigensolvers/rmmdiis.py
-index 7d60553..d182713 100644
---- a/gpaw/eigensolvers/rmmdiis.py
-+++ b/gpaw/eigensolvers/rmmdiis.py
-@@ -299,23 +299,6 @@ class RMMDIIS(Eigensolver):
- P_axi, kpt.eps_n[n_x], R_xG, n_x,
- calculate_change=True)
- self.timer.stop('Calculate residuals')
-- self.timer.start('Calculate errors')
-- errors_new_x = np.zeros(B)
-- # errors_x[:] = 0.0
-- for n in range(n1, n2):
-- if kpt.f_n is None:
-- weight = kpt.weight
-- else:
-- weight = kpt.f_n[n]
-- if self.nbands_converge != 'occupied':
-- if wfs.bd.global_index(n) < self.nbands_converge:
-- weight = kpt.weight
-- else:
-- weight = 0.0
-- errors_new_x[n-n1] += weight * integrate(R_xG[n - n1],
-- R_xG[n - n1])
-- comm.sum(errors_x)
-- self.timer.stop('Calculate errors')
-
- self.timer.stop('DIIS step')
- # Final trial step
diff --git a/gpaw/build/examples/juwels/python/build-extra.sh b/gpaw/build/examples/juwels/python/build-extra.sh
deleted file mode 100644
index 8ab157293cc7e8b382f41d235d1ad62d87ee6b5b..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/python/build-extra.sh
+++ /dev/null
@@ -1,88 +0,0 @@
-### Python (extra) modules installation script for JUWELS
-### uses PYTHONUSERBASE to bundle all modules into a separate location
-### away from the base python installation
-
-# load Python
-source $PYTHONHOME/load.sh
-
-# bundle ID (e.g. time of release) (modify if needed)
-bundle=2016-06
-
-# version numbers (modify if needed)
-numpy_version=1.10.4
-scipy_version=0.17.1
-ase_version=3.11.0
-pycuda_version=2017.1.1
-libxc_version=2.1.3
-
-# installation directory (modify!)
-tgt=$PYTHONHOME/bundle/$bundle
-
-# setup build environment
-export CFLAGS="-fPIC $CFLAGS"
-export FFLAGS="-fPIC $FFLAGS"
-
-# use --user to install modules
-export PYTHONUSERBASE=$tgt
-mkdir -p $PYTHONUSERBASE/lib/python2.7/site-packages
-
-# build in a separate directory
-mkdir bundle-$bundle
-cd bundle-$bundle
-
-# cython + mpi4py
-pip install --user cython
-pip install --user mpi4py
-
-# numpy
-git clone git://github.com/numpy/numpy.git numpy-$numpy_version
-cd numpy-$numpy_version
-git checkout v$numpy_version
-sed -e "s||$MKLROOT|g" ../../setup/juwels-mkl.cfg > site.cfg
-sed -e "s||$FFLAGS|g" ../../setup/patch-intel-fcompiler.diff > patch-intel-fcompiler.diff
-sed -e "s||$CFLAGS|g" ../../setup/patch-intel-ccompiler.diff > patch-intel-ccompiler.diff
-patch numpy/distutils/fcompiler/intel.py patch-intel-fcompiler.diff
-patch numpy/distutils/intelccompiler.py patch-intel-ccompiler.diff
-python setup.py build -j 4 install --user 2>&1 | tee loki-inst
-cd ..
-
-# scipy
-git clone git://github.com/scipy/scipy.git scipy-$scipy_version
-cd scipy-$scipy_version
-git checkout v$scipy_version
-python setup.py build -j 4 install --user 2>&1 | tee loki-inst
-cd ..
-
-# ase
-git clone https://gitlab.com/ase/ase.git ase-$ase_version
-cd ase-$ase_version
-git checkout $ase_version
-python setup.py install --user 2>&1 | tee loki-inst
-cd ..
-
-# libxc
-tar xvfz ~/src/libxc-${libxc_version}.tar.gz
-cd libxc-$libxc_version
-./configure --prefix=$PYTHONUSERBASE --enable-shared | tee loki-conf
-make | tee loki-make
-make install | tee loki-inst
-export LD_LIBRARY_PATH=$PYTHONUSERBASE/lib:$LD_LIBRARY_PATH
-cd ..
-
-# pycuda
-pip install --user pycuda==$pycuda_version
-
-# go back to the main build directory
-cd ..
-
-# if this is the first bundle, use it as default
-if [ ! -e $PYTHONHOME/bundle/default ]
-then
- cd $PYTHONHOME/bundle
- ln -s $bundle default
- cd -
-fi
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/juwels/python/build.sh b/gpaw/build/examples/juwels/python/build.sh
deleted file mode 100644
index 252f31e4dda62c25173c28995d925d520c69b6ba..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/python/build.sh
+++ /dev/null
@@ -1,39 +0,0 @@
-### Python installation script for JUWELS
-### uses --prefix to set a custom installation directory
-
-# version numbers (modify if needed)
-python_version=2.7.13
-
-# installation directory (modify!)
-tgt=$HOME/lib/python-2019-01
-
-# setup build environment
-module load CUDA/9.2.88
-module load Intel/2019.0.117-GCC-7.3.0
-module load IntelMPI/2018.4.274
-module load imkl/2019.0.117
-export CC=icc
-export CFLAGS='-O2 -xAVX2 -axCORE-AVX512'
-export CXX=icpc
-export FC=ifort
-export F90=$FC
-export FFLAGS=$CFLAGS
-
-# python
-git clone https://github.com/python/cpython.git python-$python_version
-cd python-$python_version
-git checkout v$python_version
-./configure --prefix=$tgt --enable-shared --disable-ipv6 --enable-unicode=ucs4 2>&1 | tee loki-conf
-make 2>&1 | tee loki-make
-make install 2>&1 | tee loki-inst
-cd ..
-sed -e "s||$tgt|g" setup/load-python.sh > $tgt/load.sh
-
-# install pip
-source $tgt/load.sh
-python -m ensurepip
-pip install --upgrade pip
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt
diff --git a/gpaw/build/examples/juwels/python/setup/juwels-mkl.cfg b/gpaw/build/examples/juwels/python/setup/juwels-mkl.cfg
deleted file mode 100644
index 3e2d3e8c391129507b6fbec06c00e3965863af55..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/python/setup/juwels-mkl.cfg
+++ /dev/null
@@ -1,5 +0,0 @@
-[mkl]
-library_dirs = /lib/intel64
-include_dirs = /include
-lapack_libs =
-mkl_libs = mkl_rt
diff --git a/gpaw/build/examples/juwels/python/setup/load-python.sh b/gpaw/build/examples/juwels/python/setup/load-python.sh
deleted file mode 100644
index 69b2033864e3bbf9f756946f78eb24f71cc4ddaa..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/python/setup/load-python.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/bin/bash
-export PYTHONHOME=
-export PYTHONPATH=$PYTHONHOME/lib
-export PATH=$PYTHONHOME/bin:$PATH
-export MANPATH=$PYTHONHOME/share/man:$MANPATH
-export LD_LIBRARY_PATH=$PYTHONHOME/lib:$LD_LIBRARY_PATH
-
-if [[ $# -gt 0 ]]
-then
- export PYTHONUSERBASE=$PYTHONHOME/bundle/$1
- export PATH=$PYTHONUSERBASE/bin:$PATH
- export LD_LIBRARY_PATH=$PYTHONUSERBASE/lib:$LD_LIBRARY_PATH
-elif [[ -e $PYTHONHOME/bundle/default ]]
-then
- export PYTHONUSERBASE=$PYTHONHOME/bundle/default
- export PATH=$PYTHONUSERBASE/bin:$PATH
- export LD_LIBRARY_PATH=$PYTHONUSERBASE/lib:$LD_LIBRARY_PATH
-fi
-if [[ -e $PYTHONUSERBASE/include/xc.h ]]
-then
- export LIBXCDIR=$PYTHONUSERBASE
-fi
diff --git a/gpaw/build/examples/juwels/python/setup/patch-intel-ccompiler.diff b/gpaw/build/examples/juwels/python/setup/patch-intel-ccompiler.diff
deleted file mode 100644
index 17266bdf36acd62c3109c25b5a99ee3adde2e8a4..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/python/setup/patch-intel-ccompiler.diff
+++ /dev/null
@@ -1,21 +0,0 @@
-diff --git a/numpy/distutils/intelccompiler.py b/numpy/distutils/intelccompiler.py
-index 20c6d2b..7bdc3d0 100644
---- a/numpy/distutils/intelccompiler.py
-+++ b/numpy/distutils/intelccompiler.py
-@@ -48,13 +48,12 @@ class IntelEM64TCCompiler(UnixCCompiler):
- A modified Intel x86_64 compiler compatible with a 64bit GCC-built Python.
- """
- compiler_type = 'intelem'
-- cc_exe = 'icc -m64'
-- cc_args = '-fPIC'
-+ cc_exe = 'icc'
-+ cc_args = '-fPIC '
-
- def __init__(self, verbose=0, dry_run=0, force=0):
- UnixCCompiler.__init__(self, verbose, dry_run, force)
-- self.cc_exe = ('icc -m64 -fPIC -fp-model strict -O3 '
-- '-fomit-frame-pointer -openmp -xSSE4.2')
-+ self.cc_exe = ('icc -fPIC ')
- compiler = self.cc_exe
- if platform.system() == 'Darwin':
- shared_flag = '-Wl,-undefined,dynamic_lookup'
diff --git a/gpaw/build/examples/juwels/python/setup/patch-intel-fcompiler.diff b/gpaw/build/examples/juwels/python/setup/patch-intel-fcompiler.diff
deleted file mode 100644
index a9dd7b07d5d89c4867fa5ef26903ffa7fc76b3dd..0000000000000000000000000000000000000000
--- a/gpaw/build/examples/juwels/python/setup/patch-intel-fcompiler.diff
+++ /dev/null
@@ -1,17 +0,0 @@
-diff --git a/numpy/distutils/fcompiler/intel.py b/numpy/distutils/fcompiler/intel.py
-index 2dd08e7..1df2a99 100644
---- a/numpy/distutils/fcompiler/intel.py
-+++ b/numpy/distutils/fcompiler/intel.py
-@@ -120,10 +120,10 @@ class IntelEM64TFCompiler(IntelFCompiler):
- return ['-fPIC']
-
- def get_flags_opt(self):
-- return ['-openmp -fp-model strict']
-+ return ['-fPIC ']
-
- def get_flags_arch(self):
-- return ['-xSSE4.2']
-+ return ['']
-
- # Is there no difference in the version string between the above compilers
- # and the Visual compilers?
diff --git a/gpaw/build/patch-rmmdiis.diff b/gpaw/build/patch-rmmdiis.diff
deleted file mode 100644
index adefa27c31dd72c87df2d3e35b8fcb56a7224dcf..0000000000000000000000000000000000000000
--- a/gpaw/build/patch-rmmdiis.diff
+++ /dev/null
@@ -1,34 +0,0 @@
-commit 761cba649d58e2d2f24c0a1e2fdad917b5929679
-Author: Martti Louhivuori
-Date: Thu May 18 10:56:07 2017 +0300
-
- Remove obsolete error calculation from DIIS step
-
-diff --git a/gpaw/eigensolvers/rmmdiis.py b/gpaw/eigensolvers/rmmdiis.py
-index 7d60553..d182713 100644
---- a/gpaw/eigensolvers/rmmdiis.py
-+++ b/gpaw/eigensolvers/rmmdiis.py
-@@ -299,23 +299,6 @@ class RMMDIIS(Eigensolver):
- P_axi, kpt.eps_n[n_x], R_xG, n_x,
- calculate_change=True)
- self.timer.stop('Calculate residuals')
-- self.timer.start('Calculate errors')
-- errors_new_x = np.zeros(B)
-- # errors_x[:] = 0.0
-- for n in range(n1, n2):
-- if kpt.f_n is None:
-- weight = kpt.weight
-- else:
-- weight = kpt.f_n[n]
-- if self.nbands_converge != 'occupied':
-- if wfs.bd.global_index(n) < self.nbands_converge:
-- weight = kpt.weight
-- else:
-- weight = 0.0
-- errors_new_x[n-n1] += weight * integrate(R_xG[n - n1],
-- R_xG[n - n1])
-- comm.sum(errors_x)
-- self.timer.stop('Calculate errors')
-
- self.timer.stop('DIIS step')
- # Final trial step
diff --git a/gpaw/build/examples/davide/gpaw-cpu/setup/patch-rmmdiis.diff b/gpaw/build/patches/patch-rmmdiis.diff
similarity index 100%
rename from gpaw/build/examples/davide/gpaw-cpu/setup/patch-rmmdiis.diff
rename to gpaw/build/patches/patch-rmmdiis.diff
diff --git a/gpaw/scripts/affinity-wrapper.sh b/gpaw/scripts/affinity-wrapper.sh
deleted file mode 100755
index 49105069191456d0fad6068862a24f250464b60b..0000000000000000000000000000000000000000
--- a/gpaw/scripts/affinity-wrapper.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/bin/bash
-# An affinity wrapper for KNCs.
-# As arguments, it expects first the number of processes per node
-# followed by the command to run (and any possible arguments to it).
-
-# get some information about the job
-ppn=$1
-shift
-rank=$PMI_RANK
-nmpi=$PMI_SIZE
-
-# echo "RANK", $PMI_RANK
-
-# number of devices in the system
-ndev=2
-
-# number of cores per device
-nphcores=61
-nphcores=$((nphcores - 1))
-
-# number of threads per physical core
-tpc=4
-
-# ranks per device
-rpd=$((ppn / ndev))
-if [ "$rpd" == "0" ]; then
- rpd=1
-fi
-
-# physical cores per device
-ncores=$((nphcores / rpd))
-
-# partition number of the current rank on its device
-partition=$((rank % rpd))
-
-# offset for the current rank
-offset=$((ncores * partition))
-
-# build core selection string
-select="${ncores}c,${tpc}t,${offset}o"
-
-# fire up the actual run
-log="affinity-`printf %03d $rank`.log"
-rm -f $log
-echo "host `hostname` rank `printf %03d $rank` - $select " |& tee -a $log
-env | grep PYMIC |& tee -a $log
-PYMIC_KMP_AFFINITY=compact,verbose PYMIC_KMP_PLACE_THREADS=$select $@ |& tee -a $log
diff --git a/gpaw/scripts/job-BSC-MareNostrum4-skylake.slurm b/gpaw/scripts/job-BSC-MareNostrum4-skylake.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..435e5c595d8003c0d2ab5079e41b47f44cca322d
--- /dev/null
+++ b/gpaw/scripts/job-BSC-MareNostrum4-skylake.slurm
@@ -0,0 +1,125 @@
+#! /bin/bash -l
+#SBARCH -N 21
+#SBATCH -n 1000 -c 1
+#SBATCH --time 2:00:00
+#SBATCH -J GPAWbench
+#SBATCH --qos prace
+#
+
+inputfile=../input.py
+benchmark_size='large'
+csv_summary="BSC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}.csv"
+module_20_1=GPAW-UEABS/20.1.0-Python38-FFTW-icc
+module_20_10=GPAW-UEABS/20.10.0-Python39-FFTW-icc
+
+# The next one is not a system module but one that was used to do the settings
+# for the project.
+module load UEABS/2.2
+
+srun_options=''
+
+echo -e "\nWorking in: $(pwd)\n"
+echo -e "Modules loaded:\n"
+module list
+echo -e "Slurm environment:\n$(env | grep SLURM_)\n"
+echo -e "\nJob script:\n"
+cat $0
+echo -e "\n\n"
+
+#
+# Check the results
+#
+function print_header {
+
+ echo '"Module", "python/gpaw/ase/numpy/scipy", "tasks", "time", "iterations", "dipole", "fermi", "energy", "check", "Job ID"' >$1
+
+}
+
+function print_results {
+
+ output=$1
+ summary=$2
+ module=$3
+
+ . ../bounds.sh
+
+ python_version=$(python -V | awk '{print $2}')
+ gpaw_version=$(srun -n 1 -c 1 python -c "import gpaw ; print( gpaw.__version__ )")
+ ase_version=$(python -c "import ase ; print( ase.__version__ )")
+ numpy_version=$(python -c "import numpy ; print( numpy.__version__ )")
+ scipy_version=$(python -c "import scipy ; print( scipy.__version__ )")
+
+ # Extract some data to report form the output file.
+ bmtime=$(grep "Total:" $output | sed -e 's/Total: *//' | cut -d " " -f 1)
+ iterations=$(grep "Converged after" $output | cut -d " " -f 3)
+ dipole=$(grep "Dipole" $output | cut -d " " -f 5 | sed -e 's/)//')
+ fermi=$(grep "Fermi level:" $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ energy=$(grep "Extrapolated: " $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ # Check the bounds
+ if (( $(bc -l <<< "(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations)") == 1 )); then iterations_ok="OK"; else iterations_ok="not OK"; fi
+ if (( $(bc -l <<< "(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole)") == 1 )); then dipole_ok="OK"; else dipole_ok="not OK"; fi
+ if (( $(bc -l <<< "(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi)") == 1 )); then fermi_ok="OK"; else fermi_ok="not OK"; fi
+ if (( $(bc -l <<< "(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)") == 1 )); then energy_ok="OK"; else energy_ok="not OK"; fi
+ compare=""
+ compare+="(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations) && "
+ compare+="(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole) && "
+ compare+="(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi) && "
+ compare+="(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)"
+ if (( $(bc -l <<< "$compare") ));
+ then
+ bounds_check="OK";
+ else
+ bounds_check="not OK"
+ fi
+ # Output to the slurm.out file
+ echo -e "\nResult information:\n" \
+ " * Time: $bmtime s\n" \
+ " * Number of iterations: $iterations (lower: $lower_iterations, upper: $upper_iterations, $iterations_ok)\n" \
+ " * Dipole (3rd component): $dipole (lower: $lower_dipole, upper: $upper_dipole, $dipole_ok)\n" \
+ " * Fermi level: $fermi (lower: $lower_fermi, upper: $upper_fermi, $fermi_ok)\n" \
+ " * Extrapolated energy: $energy (lower: $lower_energy, upper: $upper_energy, $energy_ok)\n" \
+ " * Boundary check: $bounds_check"
+ # Output to the summary spreadsheet
+ echo "\"$module\", \"$python_version/$gpaw_version/$ase_version/$numpy_version/$scipy_version\"," \
+ "\"$SLURM_NTASKS\", \"$bmtime\", \"$iterations\", \"$dipole\", \"$fermi\", \"$energy\", \"$bounds_check\", \"$SLURM_JOB_ID\"" >> $summary
+
+}
+
+#
+# Running with GPAW 20.1.0
+#
+
+print_header $csv_summary
+
+module load $module_20_1
+
+echo -e "\n\nStarting GPAW for $module_20_1\n"
+
+srun $srun_options gpaw python $inputfile
+
+echo -e "\n\nGPAW terminated\n"
+
+print_results output.txt $csv_summary $module_20_1
+
+mv output.txt BSC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.1.0.txt
+
+#
+# Running with GPAW 20.10.0
+#
+
+module load $module_20_10
+
+echo -e "\n\n"
+
+echo -e "\n\nStarting GPAW for $module_20_10\n"
+
+srun $srun_options gpaw python $inputfile
+
+echo -e "\n\nGPAW terminated\n"
+
+print_results output.txt $csv_summary $module_20_10
+
+mv output.txt BSC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.10.0.txt
+
+
+
diff --git a/gpaw/scripts/job-HLRS-Hawk-rome.pbs b/gpaw/scripts/job-HLRS-Hawk-rome.pbs
new file mode 100644
index 0000000000000000000000000000000000000000..2d1349a73cc7c89d2b66bc3d735704efc8b09d31
--- /dev/null
+++ b/gpaw/scripts/job-HLRS-Hawk-rome.pbs
@@ -0,0 +1,142 @@
+#! /bin/bash -l
+#PBS -l select=8:node_type=rome:mpiprocs=128
+#PBS -l walltime=1:00:00
+#PBS -N GPPAWbench
+
+numranks=1000
+inputfile=../../input.py
+benchmark_size='large'
+csv_summary="HLRS_${benchmark_size}_${PBS_JOBID}_${numranks}.csv"
+module_20_1=GPAW-UEABS/20.1.0-Python38-FFTW-icc
+module_20_10=GPAW-UEABS/20.10.0-Python39-FFTW-icc
+
+cd $PBS_O_WORKDIR
+
+compiler_module='intel/19.1.3'
+mpi_module='mpt/2.23'
+math_module='mkl/19.1.0'
+
+bounds="$(dirname ${inputfile})/bounds.sh"
+
+# The first module is not a system module but one that was used to do the settings
+# for the project.
+module load UEABS
+module load $compiler_module
+module load $mpi_module
+module load $math_module
+
+export MKL_DEBUG_CPU_TYPE=5
+export OMP_NUM_THREADS=1
+
+echo -e "\nWorking in: $(pwd)\n"
+echo -e "Modules loaded:\n"
+module list
+echo -e "PBS environment:\n$(env | grep PBS_)\n"
+echo -e "\nJob script:\n"
+cat $0
+echo -e "\n\n"
+
+#
+# Check the results
+#
+function print_header {
+
+ echo '"Module", "python/gpaw/ase/numpy/scipy", "tasks", "time", "iterations", "dipole", "fermi", "energy", "check", "Job ID"' >$1
+
+}
+
+function print_results {
+
+ output=$1
+ summary=$2
+ module=$3
+ bounds=$4
+
+ #source ${bounds}
+ source ../../bounds.sh
+
+ python_version=$(python -V | awk '{print $2}')
+ gpaw_version=$(python -c "import gpaw ; print( gpaw.__version__ )")
+ ase_version=$(python -c "import ase ; print( ase.__version__ )")
+ numpy_version=$(python -c "import numpy ; print( numpy.__version__ )")
+ scipy_version=$(python -c "import scipy ; print( scipy.__version__ )")
+
+ # Extract some data to report form the output file.
+ bmtime=$(grep "Total:" $output | sed -e 's/Total: *//' | cut -d " " -f 1)
+ iterations=$(grep "Converged after" $output | cut -d " " -f 3)
+ dipole=$(grep "Dipole" $output | cut -d " " -f 5 | sed -e 's/)//')
+ fermi=$(grep "Fermi level:" $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ energy=$(grep "Extrapolated: " $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ # Check the bounds
+ if (( $(bc -l <<< "(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations)") == 1 )); then iterations_ok="OK"; else iterations_ok="not OK"; fi
+ if (( $(bc -l <<< "(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole)") == 1 )); then dipole_ok="OK"; else dipole_ok="not OK"; fi
+ if (( $(bc -l <<< "(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi)") == 1 )); then fermi_ok="OK"; else fermi_ok="not OK"; fi
+ if (( $(bc -l <<< "(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)") == 1 )); then energy_ok="OK"; else energy_ok="not OK"; fi
+ compare=""
+ compare+="(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations) && "
+ compare+="(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole) && "
+ compare+="(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi) && "
+ compare+="(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)"
+ if (( $(bc -l <<< "$compare") ));
+ then
+ bounds_check="OK";
+ else
+ bounds_check="not OK"
+ fi
+ # Output to the PBS output file
+ echo -e "\nResult information:\n" \
+ " * Time: $bmtime s\n" \
+ " * Number of iterations: $iterations (lower: $lower_iterations, upper: $upper_iterations, $iterations_ok)\n" \
+ " * Dipole (3rd component): $dipole (lower: $lower_dipole, upper: $upper_dipole, $dipole_ok)\n" \
+ " * Fermi level: $fermi (lower: $lower_fermi, upper: $upper_fermi, $fermi_ok)\n" \
+ " * Extrapolated energy: $energy (lower: $lower_energy, upper: $upper_energy, $energy_ok)\n" \
+ " * Boundary check: $bounds_check"
+ # Output to the summary spreadsheet
+ echo "\"$module\", \"$python_version/$gpaw_version/$ase_version/$numpy_version/$scipy_version\"," \
+ "\"$numranks\", \"$bmtime\", \"$iterations\", \"$dipole\", \"$fermi\", \"$energy\", \"$bounds_check\", \"$PBS_JOBID\"" >> $summary
+
+}
+
+#
+# Running with GPAW 20.1.0
+#
+
+print_header $csv_summary
+
+module purge
+module load UEABS/2.2
+module load $module_20_1
+echo -e "\n\nGPAW run with $module_20_1\nModules loaded:\n"
+module list 2>&1
+
+echo -e "\n\nStarting GPAW...\n"
+
+mpirun -n $numranks gpaw python $inputfile
+
+echo -e "\nGPAW ended, checking results...\n"
+
+print_results output.txt $csv_summary $module_20_1
+
+mv output.txt HLRS_${benchmark_size}_${PBS_JOBID}_${numranks}_20.1.0.txt
+
+#
+# Running with GPAW 20.10.0
+#
+
+module purge
+module load UEABS/2.2
+module load $module_20_10
+echo -e "\n\nGPAW run with $module_20_10\nModules loaded:\n"
+module list 2>&1
+
+echo -e "\n\nStarting GPAW...\n"
+echo -e "\n\n"
+
+mpirun -n $numranks gpaw python $inputfile $bounds
+
+echo -e "\nGPAW ended, checking results...\n"
+
+print_results output.txt $csv_summary $module_20_10 $bounds
+
+mv output.txt HLRS_${benchmark_size}_${PBS_JOBID}_${numranks}_20.10.0.txt
+
diff --git a/gpaw/scripts/job-JSC-JUWELS-skylake.slurm b/gpaw/scripts/job-JSC-JUWELS-skylake.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..bf58089c216b764176bf1d20501b43634e52c86f
--- /dev/null
+++ b/gpaw/scripts/job-JSC-JUWELS-skylake.slurm
@@ -0,0 +1,127 @@
+#! /bin/bash -l
+#SBATCH -A prpb101
+#SBARCH -N 21
+#SBATCH -n 1000 -c 1
+#SBATCH --time 1:00:00
+#SBATCH -J GPAWbench_large
+#SBATCH -p batch
+#SBATCH --hint=nomultithread
+#SBATCH -o %x-%j.out
+#
+
+inputfile=../input.py
+benchmark_size='large'
+csv_summary="JSC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}.csv"
+module_20_1=GPAW-UEABS/20.1.0-Python38-FFTW-icc
+module_20_10=GPAW-UEABS/20.10.0-Python39-FFTW-icc
+
+module load Intel/2021.2.0-GCC-10.3.0
+module load IntelMPI/2021.2.0
+
+echo -e "\nWorking in: $(pwd)\n"
+echo -e "Modules loaded:\n"
+module list
+echo -e "Slurm environment:\n$(env | grep SLURM_)\n"
+echo -e "\nJob script:\n"
+cat $0
+echo -e "\n\n"
+
+
+#
+# Check the results
+#
+function print_header {
+
+ echo '"Module", "python/gpaw/ase/numpy/scipy", "tasks", "time", "iterations", "dipole", "fermi", "energy", "check", "Job ID"' >$1
+
+}
+
+function print_results {
+
+ output=$1
+ summary=$2
+ module=$3
+
+ . ../bounds.sh
+
+ python_version=$(python -V | awk '{print $2}')
+ gpaw_version=$(srun -n 1 -c 1 python -c "import gpaw ; print( gpaw.__version__ )")
+ ase_version=$(python -c "import ase ; print( ase.__version__ )")
+ numpy_version=$(python -c "import numpy ; print( numpy.__version__ )")
+ scipy_version=$(python -c "import scipy ; print( scipy.__version__ )")
+
+ # Extract some data to report form the output file.
+ bmtime=$(grep "Total:" $output | sed -e 's/Total: *//' | cut -d " " -f 1)
+ iterations=$(grep "Converged after" $output | cut -d " " -f 3)
+ dipole=$(grep "Dipole" $output | cut -d " " -f 5 | sed -e 's/)//')
+ fermi=$(grep "Fermi level:" $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ energy=$(grep "Extrapolated: " $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ # Check the bounds
+ if (( $(bc -l <<< "(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations)") == 1 )); then iterations_ok="OK"; else iterations_ok="not OK"; fi
+ if (( $(bc -l <<< "(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole)") == 1 )); then dipole_ok="OK"; else dipole_ok="not OK"; fi
+ if (( $(bc -l <<< "(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi)") == 1 )); then fermi_ok="OK"; else fermi_ok="not OK"; fi
+ if (( $(bc -l <<< "(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)") == 1 )); then energy_ok="OK"; else energy_ok="not OK"; fi
+ compare=""
+ compare+="(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations) && "
+ compare+="(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole) && "
+ compare+="(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi) && "
+ compare+="(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)"
+ if (( $(bc -l <<< "$compare") ));
+ then
+ bounds_check="OK";
+ else
+ bounds_check="not OK"
+ fi
+ # Output to the slurm.out file
+ echo -e "\nResult information:\n" \
+ " * Time: $bmtime s\n" \
+ " * Number of iterations: $iterations (lower: $lower_iterations, upper: $upper_iterations, $iterations_ok)\n" \
+ " * Dipole (3rd component): $dipole (lower: $lower_dipole, upper: $upper_dipole, $dipole_ok)\n" \
+ " * Fermi level: $fermi (lower: $lower_fermi, upper: $upper_fermi, $fermi_ok)\n" \
+ " * Extrapolated energy: $energy (lower: $lower_energy, upper: $upper_energy, $energy_ok)\n" \
+ " * Boundary check: $bounds_check"
+ # Output to the summary spreadsheet
+ echo "\"$module\", \"$python_version/$gpaw_version/$ase_version/$numpy_version/$scipy_version\"," \
+ "\"$SLURM_NTASKS\", \"$bmtime\", \"$iterations\", \"$dipole\", \"$fermi\", \"$energy\", \"$bounds_check\", \"$SLURM_JOB_ID\"" >> $summary
+
+}
+
+#
+# Running with GPAW 20.1.0
+#
+
+print_header $csv_summary
+
+module purge
+# UEABS/2.2 is not a system module but one of our own to basically do some settings
+# needed for the project (and to adjust the MODULEPATH). It is not needed if
+# $module_20_1 (generated by the build scripts) is in the MODULEPATH
+module load UEABS/2.2
+module load $module_20_1
+
+echo -e "\n\n"
+
+srun gpaw python $inputfile
+
+print_results output.txt $csv_summary $module_20_1
+
+mv output.txt JSC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.1.0.txt
+
+#
+# Running with GPAW 20.10.0
+#
+
+module purge
+module load UEABS/2.2
+module load $module_20_10
+
+echo -e "\n\n"
+
+srun gpaw python $inputfile
+
+print_results output.txt $csv_summary $module_20_10
+
+mv output.txt JSC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.10.0.txt
+
+
+
diff --git a/gpaw/scripts/job-LRZ-SuperMUCng-skylake.slurm b/gpaw/scripts/job-LRZ-SuperMUCng-skylake.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..d1e284d9c7f29993b86e3f72f15a4b778beb94f5
--- /dev/null
+++ b/gpaw/scripts/job-LRZ-SuperMUCng-skylake.slurm
@@ -0,0 +1,143 @@
+#! /bin/bash -l
+#SBATCH -A pn73ye
+#SBARCH -N 21
+#SBATCH -n 1000 -c 1
+#SBATCH --time 40:00
+#SBATCH -J GPAWbench
+#SBATCH -p general
+#SBATCH --hint=nomultithread
+#SBATCH -o %x-%j.out
+#
+
+inputfile=../input.py
+benchmark_size='large'
+csv_summary="LRZ_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}.csv"
+module_20_1=GPAW-UEABS/20.1.0-Python38-FFTW-icc
+module_20_10=GPAW-UEABS/20.10.0-Python39-FFTW-icc
+
+module unload intel-mkl
+module unload intel-mpi
+module unload intel
+
+# UEABS/2.2 is not a system module but one of our own to basically do some settings
+# needed for the project (and to adjust the MODULEPATH). It is not needed if
+# $module_20_1 (generated by the build scripts) is in the MODULEPATH
+module load UEABS/2.2
+
+echo -e "\nWorking in: $(pwd)\n"
+echo -e "Modules loaded:\n"
+module list
+echo -e "Slurm environment:\n$(env | grep SLURM_)\n"
+echo -e "\nJob script:\n"
+cat $0
+echo -e "\n\n"
+echo -e "\nCalled script:\n"
+cat ../../LRZ_run_GPAW.slurm
+echo -e "\n\n"
+
+if (( $SLURM_NTASKS <= 512 ))
+then
+ #srun --distribution=block:block /hppfs/work/pn73ye/di46ras/UEABS/Run/TEST-JOB/mpi_hello.exe
+ srun /hppfs/work/pn73ye/di46ras/UEABS/Run/TEST-JOB/mpi_hello.exe
+fi
+
+
+#
+# Check the results
+#
+function print_header {
+
+ echo '"Module", "python/gpaw/ase/numpy/scipy", "tasks", "time", "iterations", "dipole", "fermi", "energy", "check", "Job ID"' >$1
+
+}
+
+function print_results {
+
+ output=$1
+ summary=$2
+ module=$3
+
+ . ../bounds.sh
+
+ python_version=$(python -V | awk '{print $2}')
+ gpaw_version=$(srun -n 1 -c 1 python -c "import gpaw ; print( gpaw.__version__ )")
+ ase_version=$(python -c "import ase ; print( ase.__version__ )")
+ numpy_version=$(python -c "import numpy ; print( numpy.__version__ )")
+ scipy_version=$(python -c "import scipy ; print( scipy.__version__ )")
+
+ # Extract some data to report form the output file.
+ bmtime=$(grep "Total:" $output | sed -e 's/Total: *//' | cut -d " " -f 1)
+ iterations=$(grep "Converged after" $output | cut -d " " -f 3)
+ dipole=$(grep "Dipole" $output | cut -d " " -f 5 | sed -e 's/)//')
+ fermi=$(grep "Fermi level:" $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ energy=$(grep "Extrapolated: " $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ # Check the bounds
+ if (( $(bc -l <<< "(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations)") == 1 )); then iterations_ok="OK"; else iterations_ok="not OK"; fi
+ if (( $(bc -l <<< "(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole)") == 1 )); then dipole_ok="OK"; else dipole_ok="not OK"; fi
+ if (( $(bc -l <<< "(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi)") == 1 )); then fermi_ok="OK"; else fermi_ok="not OK"; fi
+ if (( $(bc -l <<< "(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)") == 1 )); then energy_ok="OK"; else energy_ok="not OK"; fi
+ compare=""
+ compare+="(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations) && "
+ compare+="(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole) && "
+ compare+="(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi) && "
+ compare+="(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)"
+ if (( $(bc -l <<< "$compare") ));
+ then
+ bounds_check="OK";
+ else
+ bounds_check="not OK"
+ fi
+ # Output to the slurm.out file
+ echo -e "\nResult information:\n" \
+ " * Time: $bmtime s\n" \
+ " * Number of iterations: $iterations (lower: $lower_iterations, upper: $upper_iterations, $iterations_ok)\n" \
+ " * Dipole (3rd component): $dipole (lower: $lower_dipole, upper: $upper_dipole, $dipole_ok)\n" \
+ " * Fermi level: $fermi (lower: $lower_fermi, upper: $upper_fermi, $fermi_ok)\n" \
+ " * Extrapolated energy: $energy (lower: $lower_energy, upper: $upper_energy, $energy_ok)\n" \
+ " * Boundary check: $bounds_check"
+ # Output to the summary spreadsheet
+ echo "\"$module\", \"$python_version/$gpaw_version/$ase_version/$numpy_version/$scipy_version\"," \
+ "\"$SLURM_NTASKS\", \"$bmtime\", \"$iterations\", \"$dipole\", \"$fermi\", \"$energy\", \"$bounds_check\", \"$SLURM_JOB_ID\"" >> $summary
+
+}
+
+#
+# Running with GPAW 20.1.0
+#
+
+print_header $csv_summary
+
+module load $module_20_1
+
+echo -e "\n\nStarting GPAW for $module_20_1\n"
+
+#srun --distribution=block:block gpaw python $inputfile
+srun gpaw python $inputfile
+
+echo -e "\n\nGPAW terminated\n"
+
+print_results output.txt $csv_summary $module_20_1
+
+mv output.txt LRZ_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.1.0.txt
+
+#
+# Running with GPAW 20.10.0
+#
+
+module load $module_20_10
+
+echo -e "\n\n"
+
+echo -e "\n\nStarting GPAW for $module_20_10\n"
+
+#srun --distribution=block:block gpaw python $inputfile
+srun gpaw python $inputfile
+
+echo -e "\n\nGPAW terminated\n"
+
+print_results output.txt $csv_summary $module_20_10
+
+mv output.txt LRZ_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.10.0.txt
+
+
+
diff --git a/gpaw/scripts/job-TGCC-irene-rome.slurm b/gpaw/scripts/job-TGCC-irene-rome.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..ede666a4995eef3991cddf9c93a098b52a5c80f8
--- /dev/null
+++ b/gpaw/scripts/job-TGCC-irene-rome.slurm
@@ -0,0 +1,130 @@
+#! /bin/bash -l
+#MSUB -A pa5772
+#MSUB -q rome
+#MSUB -Q normal
+#MSUB -n 1024
+#MSUB -c 1
+#MSUB -T 5000
+#MSUB -r GPAWbench
+#
+
+inputfile=../input.py
+boundsfile=../bounds.sh
+benchmark_size='large'
+csv_summary="TGCC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}.csv"
+module_20_1=GPAW-UEABS/20.1.0-Python38-FFTW-icc
+module_20_10=GPAW-UEABS/20.10.0-Python39-FFTW-icc
+
+cd ${BRIDGE_MSUB_PWD}
+
+module purge
+# UEABS/2.2 is not a system module but one of our own to basically do some settings
+# needed for the project (and to adjust the MODULEPATH). It is not needed if
+# $module_20_1 (generated by the build scripts) is in the MODULEPATH
+module load UEABS/2.2
+
+echo -e "\nWorking in: $(pwd)\n"
+echo -e "Modules loaded:\n"
+module list
+echo -e "Slurm environment:\n$(env | grep SLURM_)\n"
+echo -e "\nJob script:\n"
+cat $0
+echo -e "\n\n"
+
+#
+# Check the results
+#
+function print_header {
+
+ echo '"Module", "python/gpaw/ase/numpy/scipy", "tasks", "time", "iterations", "dipole", "fermi", "energy", "check", "Job ID"' >$1
+
+}
+
+function print_results {
+
+ output="$1"
+ summary="$2"
+ module="$3"
+ bounds="$4"
+
+ . $bounds
+
+ python_version=$(python -V | awk '{print $2}')
+ gpaw_version=$(srun -n 1 -c 1 python -c "import gpaw ; print( gpaw.__version__ )")
+ ase_version=$(python -c "import ase ; print( ase.__version__ )")
+ numpy_version=$(python -c "import numpy ; print( numpy.__version__ )")
+ scipy_version=$(python -c "import scipy ; print( scipy.__version__ )")
+
+ # Extract some data to report form the output file.
+ bmtime=$(grep "Total:" $output | sed -e 's/Total: *//' | cut -d " " -f 1)
+ iterations=$(grep "Converged after" $output | cut -d " " -f 3)
+ dipole=$(grep "Dipole" $output | cut -d " " -f 5 | sed -e 's/)//')
+ fermi=$(grep "Fermi level:" $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ energy=$(grep "Extrapolated: " $output | cut -d ":" -f 2 | sed -e 's/ //g')
+ # Check the bounds
+ if (( $(bc -l <<< "(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations)") == 1 )); then iterations_ok="OK"; else iterations_ok="not OK"; fi
+ if (( $(bc -l <<< "(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole)") == 1 )); then dipole_ok="OK"; else dipole_ok="not OK"; fi
+ if (( $(bc -l <<< "(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi)") == 1 )); then fermi_ok="OK"; else fermi_ok="not OK"; fi
+ if (( $(bc -l <<< "(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)") == 1 )); then energy_ok="OK"; else energy_ok="not OK"; fi
+ compare=""
+ compare+="(($iterations-0) >= $lower_iterations) && (($iterations-0) <= $upper_iterations) && "
+ compare+="(($dipole-0) >= $lower_dipole) && (($dipole-0) <= $upper_dipole) && "
+ compare+="(($fermi-0) >= $lower_fermi) && (($fermi-0) <= $upper_fermi) && "
+ compare+="(($energy-0) >= $lower_energy) && (($energy-0) <= $upper_energy)"
+ if (( $(bc -l <<< "$compare") ));
+ then
+ bounds_check="OK";
+ else
+ bounds_check="not OK"
+ fi
+ # Output to the slurm.out file
+ echo -e "\nResult information:\n" \
+ " * Time: $bmtime s\n" \
+ " * Number of iterations: $iterations (lower: $lower_iterations, upper: $upper_iterations, $iterations_ok)\n" \
+ " * Dipole (3rd component): $dipole (lower: $lower_dipole, upper: $upper_dipole, $dipole_ok)\n" \
+ " * Fermi level: $fermi (lower: $lower_fermi, upper: $upper_fermi, $fermi_ok)\n" \
+ " * Extrapolated energy: $energy (lower: $lower_energy, upper: $upper_energy, $energy_ok)\n" \
+ " * Boundary check: $bounds_check"
+ # Output to the summary spreadsheet
+ echo "\"$module\", \"$python_version/$gpaw_version/$ase_version/$numpy_version/$scipy_version\"," \
+ "\"$SLURM_NTASKS\", \"$bmtime\", \"$iterations\", \"$dipole\", \"$fermi\", \"$energy\", \"$bounds_check\", \"$SLURM_JOB_ID\"" >> $summary
+
+}
+
+#
+# Running with GPAW 20.1.0
+#
+
+print_header $csv_summary
+
+module load $module_20_1
+
+echo -e "\n\nStarting GPAW for $module_20_1\n"
+
+ccc_mprun gpaw python $inputfile
+
+echo -e "\n\nGPAW terminated\n"
+
+print_results output.txt $csv_summary $module_20_1 $boundsfile
+
+mv output.txt TGCC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.1.0.txt
+
+#
+# Running with GPAW 20.10.0
+#
+
+module load $module_20_10
+
+echo -e "\n\n"
+
+echo -e "\n\nStarting GPAW for $module_20_10\n"
+
+ccc_mprun gpaw python $inputfile
+
+echo -e "\n\nGPAW terminated\n"
+
+print_results output.txt $csv_summary $module_20_10 $boundsfile
+
+mv output.txt TGCC_${benchmark_size}_${SLURM_JOB_ID}_${SLURM_NTASKS}_20.10.0.txt
+
+
diff --git a/gpaw/scripts/job-davide.sh b/gpaw/scripts/job-davide.sh
deleted file mode 100644
index 5d145983f9f78765fadc2e74076b732a053bd9b5..0000000000000000000000000000000000000000
--- a/gpaw/scripts/job-davide.sh
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/bin/bash
-#SBATCH -J 4x4
-#SBATCH -p dvd_usr_prod
-#SBATCH --nodes=4
-#SBATCH --ntasks-per-node=4
-#SBATCH --cpus-per-task=4
-#SBATCH --time=00:30:00
-#SBATCH --account=
-#SBATCH --gres=gpu:tesla:4
-#SBATCH --exclusive
-
-cd $SLURM_SUBMIT_DIR
-
-source $CINECA_SCRATCH/lib/gpaw-cuda/load.sh
-
-srun gpaw-python input.py
-
diff --git a/gpaw/scripts/job-juwels.sh b/gpaw/scripts/job-juwels.sh
deleted file mode 100644
index ff42a63d7c3ea428f0400a745f6e5e91198ab713..0000000000000000000000000000000000000000
--- a/gpaw/scripts/job-juwels.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-#!/bin/bash -x
-#SBATCH -J 4x48
-#SBATCH --account=
-#SBATCH --nodes=4
-#SBATCH --ntasks-per-node=48
-#SBATCH --time=00:30:00
-#SBATCH --partition=batch
-
-source $HOME/lib/gpaw-1.1.0/load.sh
-
-srun gpaw-python input.py
-
diff --git a/gpaw/scripts/job-marenostrum.sh b/gpaw/scripts/job-marenostrum.sh
deleted file mode 100644
index 98303237358880fd9fa28bbbd16889b1136f63d8..0000000000000000000000000000000000000000
--- a/gpaw/scripts/job-marenostrum.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/bash -x
-#SBATCH -J 4x48
-#SBATCH --nodes=4
-#SBATCH --ntasks-per-node=48
-#SBATCH --time=00:30:00
-#SBATCH --qos=prace
-
-source $HOME/project/lib/gpaw-1.1.0/load.sh
-
-srun gpaw-python input.py
-
diff --git a/specfem3d/compile.sh b/specfem3d/compile.sh
index d99d3e54b1711022d07d8b5416c09ace668a232e..437ba48c47699b99db03069e10b0c04f0522a0c3 100755
--- a/specfem3d/compile.sh
+++ b/specfem3d/compile.sh
@@ -17,9 +17,13 @@ echo " - deepsdv "
echo " - hazelhen "
echo " - vega-gpu"
echo " - vega-cpu"
-read machine
+echo " - marconi100 "
+echo " - supermuc-ng "
+read machine
+export NEX_XI=384 #384 #416 #352 #320 #288 #256 #224 #192 #160 #128 #96 #288
+export NPROC_XI=4
source ./env/env_${machine}
Untar(){
@@ -70,8 +74,10 @@ Install(){
mkdir -p $install_dir
mv $install_dir/../specfem3d_globe $install_dir/.
cp test_cases/SPECFEM3D_TestCaseA/* $install_dir/specfem3d_globe/DATA/.
- sed -i s/"NEX_XI = 384"/"NEX_XI = $NEX_XI"/g $install_dir/specfem3d_globe/DATA/Par_file
- sed -i s/"NEX_ETA = 384"/"NEX_ETA = $NEX_XI"/g $install_dir/specfem3d_globe/DATA/Par_file
+ sed -i s/"NEX_XI = 384"/"NEX_XI = $NEX_XI"/g $install_dir/specfem3d_globe/DATA/Par_file
+ sed -i s/"NEX_ETA = 384"/"NEX_ETA = $NEX_XI"/g $install_dir/specfem3d_globe/DATA/Par_file
+ sed -i s/"NPROC_XI = 4"/"NPROC_XI = $NPROC_XI"/g $install_dir/specfem3d_globe/DATA/Par_file
+ sed -i s/"NPROC_ETA = 4"/"NPROC_ETA = $NPROC_XI"/g $install_dir/specfem3d_globe/DATA/Par_file
elif [ $answer = "3" ]; then
echo "===> Copy test case B"
export install_dir=$install_dir/TestCaseB
@@ -100,6 +106,12 @@ Install(){
sed -i s/"GPU_MODE = .false."/"GPU_MODE = .true."/g $install_dir/specfem3d_globe/DATA/Par_file
sed -i s/"GPU_DEVICE = Tesla"/"GPU_DEVICE = *"/g $install_dir/specfem3d_globe/DATA/Par_file
./configure --build=ppc64 --with-cuda=cuda8
+ elif [ $machine = "marconi100" ]; then
+ echo "Configure for Marconi100 GPU partion"
+ sed -i s/"GPU_MODE = .false."/"GPU_MODE = .true."/g $install_dir/specfem3d_globe/DATA/Par_file
+ sed -i s/"GPU_DEVICE = Tesla"/"GPU_DEVICE = *"/g $install_dir/specfem3d_globe/DATA/Par_file
+ ./configure --build=ppc64 --with-cuda=cuda8
+ sed -i s/"O4"/"O3"/g Makefile #-O4 is not supported with option -qoffload
else
echo "Configure for CPU only system"
./configure --enable-openmp
@@ -144,7 +156,7 @@ Clean(){
Deploy(){
echo "install_dir ="$install_dir
- if [ $machine = "occigen" ] || [ $machine = "marenostrum" ] || [ $machine = "marconi-knl" ] || [ $machine = "daint-cpu-only" ] || [ $machine = "daint-gpu" ] || [ $machine = "davide" ] || [ $machine = "juwels" ] || [ $machine = "juwels-booster" ] || [ $machine = "irene-skl" ] || [ $machine = "irene-knl" ] || [ $machine = "dibona" ] || [ $machine = "frioul" ] || [ $machine = "deepsdv" ] || [ $machine = "hazelhen" ] || [ $machine = "vega-cpu" ] || [ $machine = "vega-gpu" ];then
+ if [ $machine = "occigen" ] || [ $machine = "marenostrum" ] || [ $machine = "marconi-knl" ] || [ $machine = "daint-cpu-only" ] || [ $machine = "daint-gpu" ] || [ $machine = "davide" ] || [ $machine = "juwels" ] || [ $machine = "juwels-booster" ] || [ $machine = "irene-skl" ] || [ $machine = "irene-knl" ] || [ $machine = "dibona" ] || [ $machine = "frioul" ] || [ $machine = "deepsdv" ] || [ $machine = "hazelhen" ] || [ $machine = "vega-cpu" ] || [ $machine = "vega-gpu" ] || [ $machine = "marconi100" ] || [ $machine = "supermuc-ng" ];then
echo "==> Install on $machine :"
mkdir -p $install_dir
Clean
diff --git a/specfem3d/env/env/env_supermuc-ng b/specfem3d/env/env/env_supermuc-ng
new file mode 100644
index 0000000000000000000000000000000000000000..9692290879539fe339b868e39b591317cdbe9bc9
--- /dev/null
+++ b/specfem3d/env/env/env_supermuc-ng
@@ -0,0 +1,14 @@
+#!/bin/bash
+module purge
+module load admin/1.0 tempdir lrz/1.0 intel-oneapi/2021.2
+module li
+
+export machine=supermuc-ng
+export software=specfem3d_globe
+export version=31octobre
+export install_dir=$WORK_pn73ve/benchmarks/$machine/$software/$version/
+export CC="mpiicc"
+export FC="mpiifort"
+export MPIFC=$FC
+export FCFLAGS=" -O3 -qopenmp -xCORE-AVX512 -mtune=skylake -ipo -no-prec-div -no-prec-sqrt -fma -qopt-zmm-usage=high -DUSE_FP32 -DOPT_STREAMS -fp-model fast=2 -traceback -mcmodel=large"
+export CFLAGS=" -O3 -qopenmp -xCORE-AVX512 -mtune=skylake -ipo -no-prec-div -no-prec-sqrt -fma -qopt-zmm-usage=high"
diff --git a/specfem3d/env/env_marconi100 b/specfem3d/env/env_marconi100
new file mode 100644
index 0000000000000000000000000000000000000000..4322d8449063269a070af5923c598579c130b361
--- /dev/null
+++ b/specfem3d/env/env_marconi100
@@ -0,0 +1,19 @@
+#!/bin/bash
+module purge
+module load profile/base
+module load xl/16.1.1--binary spectrum_mpi/10.3.1--binary cuda/11.0
+
+export machine=marconi100
+export software=specfem3d_globe
+export version=31octobre
+export install_dir=$CINECA_SCRATCH/benchmarks/$machine/$software/$version
+
+# Power9
+export CC="mpixlc"
+export FC="mpixlf"
+export MPIFC=mpixlf
+#DEBUG="-qcheck -g -qsigtrap" # -qstackprotect=all -> internal compiler error : https://www.ibm.com/support/pages/node/722473
+#FLAGS_CHECK="-g -qfullpath -O2 -qsave -qstrict -qtune=qp -qarch=qp -qcache=auto -qhalt=w -qfree=f90 -qsuffix=f=f90 -qlanglvl=95pure -Q -Q+rank,swap_all -Wl,-relax"
+#FLAGS_CHECK="-g -qfullpath -qsave -qtune=auto -qarch=auto -qcache=auto -qhalt=w -qfree=f90 -qsuffix=f=f90 -qlanglvl=2003pure"
+export FCFLAGS="-O3 -DUSE_FP32 -DOPT_STREAMS -qpic $FLAGS_CHECK $DEBUG" #-mcmodel=large -qoffload
+export CFLAGS="-O3 -qpic $FLAGS_CHECK $DEBUG" #-qoffload
diff --git a/specfem3d/env/env_vega-cpu b/specfem3d/env/env_vega-cpu
index e1c48990846de280c6e2ea1c1593c694164564d3..9b31b68aeb2c3e539bf914ce62de8d69e7cf953b 100644
--- a/specfem3d/env/env_vega-cpu
+++ b/specfem3d/env/env_vega-cpu
@@ -5,6 +5,8 @@ export machine=vega-cpu
export software=specfem3d_globe
export version=31octobre
+export NEX_XI=384 #384 #416 #352 #320 #288 #256 #224 #192 #160 #128 #96 #288
+export NPROC_XI=4
MYSCRATCH=/exa5/scratch/user/eucedricj
export install_dir=$MYSCRATCH//benchmarks/$machine/$software/$version/cpu-znver2/
echo "install_dir = " $install_dir
@@ -14,17 +16,5 @@ export FC=mpif90 #mpif90 #mpiifort
export MPIFC=$FC
# gnu
-#Test done with those flags 01/10/21
export FCFLAGS=" -O3 -fopenmp -march=znver2 -mtune=znver2 -flto -funroll-all-loops -ffast-math -mfma -mavx2 -m3dnow -fomit-frame-pointer -DUSE_FP32 -DOPT_STREAMS -mcmodel=large"
export CFLAGS=" -O3 -fopenmp -march=znver2 -mtune=znver2 -flto -funroll-all-loops -ffast-math -mfma -mavx2 -m3dnow -fomit-frame-pointer"
-# Test Ok but not better than upper flags
-#export FCFLAGS=" -O3 -fopenmp -march=znver2 -flto -ffast-math -mfma -mavx2 -fomit-frame-pointer -DUSE_FP32 -DOPT_STREAMS -mcmodel=large"
-#export CFLAGS=" -O3 -fopenmp -march=znver2 -flto -ffast-math -mfma -mavx2 -fomit-frame-pointer"
-
-#INTEL
-#export FCFLAGS=" -O3 -qopenmp -march=core-avx2 -mtune=core-avx2 -ipo -no-prec-div -no-prec-sqrt -fma -qopt-prefetch -use-intel-optimized-headers -DUSE_FP32 -DOPT_STREAMS -fp-model fast=2 -mcmodel=large"
-#export CFLAGS=" -O3 -qopenmp -march=core-avx2 -mtune=core-avx2 -ipo -no-prec-div -no-prec-sqrt -fma -qopt-prefetch -use-intel-optimized-headers"
-
-# aocc
-#export FCFLAGS="-O3 -march=znver2 -mtune=znver2 -fopenmp -flto -DUSE_FP32 -DOPT_STREAMS -mcmodel=large"
-#export CFLAGS=" -O3 -march=znver2 -mtune=znver2 -fopenmp -flto"
diff --git a/specfem3d/env/env_vega-gpu b/specfem3d/env/env_vega-gpu
index 08cae92a23cc748dfc95a42320a644630b433919..166db4b196294d85ad432f160f5e7777dc8496c1 100644
--- a/specfem3d/env/env_vega-gpu
+++ b/specfem3d/env/env_vega-gpu
@@ -1,19 +1,15 @@
#!/bin/bash
-
module purge
-#module load GCC/10.2.0 OpenMPI/4.0.5-gcccuda-2020b #compilaton error...
-#module load GCC/8.3.0 OpenMPI/3.1.4-gcccuda-2019b
-#module load oneapi/compiler/2021.2.0 openmpi/intel/4.0.5.2 CUDA/10.1.105 #oneapi/intel_ipp_intel64/2021.2.0 #oneapi/intel_ipp_ia32/2021.2.0
-#module load GCC/10.2.0 openmpi/gnu/4.0.5.2 CUDA/11.1.1-GCC-10.2.0 #Compilation Error: Rank mismatch between actual argument at (1) and actual argument at (2)..
module load GCC/9.3.0 openmpi/gnu/4.0.5.2 CUDA/11.0.2-GCC-9.3.0
export machine=vega-gpu
export software=specfem3d_globe
export version=31octobre
-MYSCRATCH=/exa5/scratch/user/eucedricj
-export NEX_XI=384 #416 #352 #320 #288 #256 #224 #192 #160 #128 #96 #288
-export install_dir=$MYSCRATCH//benchmarks/$machine/$software/$version/strong-scaling/$NEX_XI
+export NEX_XI=384 #384 #416 #352 #320 #288 #256 #224 #192 #160 #128 #96 #288
+export NPROC_XI=4
+MYSCRATCH=/exa5/scratch/user/eucedricj
+export install_dir=$MYSCRATCH//benchmarks/$machine/$software/$version
echo "install_dir = " $install_dir
export CC=mpicc #mpiicc
@@ -26,11 +22,3 @@ export CFLAGS=" -O3 -flto -march=znver2 -mtune=znver2 -ffast-math -mfma -mavx2 -
export CUDA_LIB=$CUDA_HOME/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/
export LDFLAGS="$LDFLAGS -lpthread"
-
-#INTEL
-#export FCFLAGS=" -O3 -qopenmp -march=core-avx2 -mtune=core-avx2 -ipo -no-prec-div -no-prec-sqrt -fma -qopt-prefetch -use-intel-optimized-headers -DUSE_FP32 -DOPT_STREAMS -fp-model fast=2 -mcmodel=large"
-#export CFLAGS=" -O3 -qopenmp -march=core-avx2 -mtune=core-avx2 -ipo -no-prec-div -no-prec-sqrt -fma -qopt-prefetch -use-intel-optimized-headers"
-
-# aocc
-#export FCFLAGS="-O3 -march=znver2 -mtune=znver2 -fopenmp -flto -DUSE_FP32 -DOPT_STREAMS -mcmodel=large"
-#export CFLAGS=" -O3 -march=znver2 -mtune=znver2 -fopenmp -flto"
diff --git a/specfem3d/job_script/job_marconi100_small_benchmark_run_to_test_more_complex_Earth.slurm b/specfem3d/job_script/job_marconi100_small_benchmark_run_to_test_more_complex_Earth.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..87f600f121cd49d62ddb817c1f6dc0b2e6b884b3
--- /dev/null
+++ b/specfem3d/job_script/job_marconi100_small_benchmark_run_to_test_more_complex_Earth.slurm
@@ -0,0 +1,75 @@
+#!/bin/bash
+#SBATCH -J Validation_case_specfem-Marconi100_small_benchmark_run_to_test_more_complex_Earth
+#SBATCH -A Ppp4x_5850
+#SBATCH -p m100_usr_prod
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=24 # 24 tasks out of 128
+#SBATCH --cpus-per-task=4
+#SBATCH --time=01:59:59
+#SBATCH --output Validation_case_specfem-Marconi100_small_benchmark_run_to_test_more_complex_Earth-xl-spectrumpi-mpirun-GPU_DEVICE-star-%j.out
+#SBATCH --gres=gpu:4 # 1 gpus per node out of 4
+##SBATCH --hint=nomultithread
+#SBATCH --exclusive
+#SBATCH --mem=246000 # memory per node out of 246000MB
+
+#set -e
+source ../env/env_marconi100
+echo "Environment used:"
+echo "================="
+grep -E -v '^(#|$)' ../env/env_marconi100
+cat job_marconi100_small_benchmark_run_to_test_more_complex_Earth.slurm
+cd $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+
+# Uncomment the 3-5 following lines if it's first time launched after the compilation:
+sed -i s/"GPU_MODE = .false."/"GPU_MODE = .true."/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/DATA/Par_file
+sed -i s/"GPU_DEVICE = Tesla"/"GPU_DEVICE = *"/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/DATA/Par_file
+sed -i s/"configure --enable-openmp"/"configure --build=ppc64 --with-cuda=cuda8"/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_this_example.sh
+#sed -i s/"mpirun -np"/"srun -n"/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_mesher_solver.bash
+sed -i s/"mpirun -np"/"mpirun -gpu -np"/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_mesher_solver.bash
+sed -i '38d' $install_dir/specfem3d_globe/Makefile
+grep configure $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_this_example.sh
+
+#export OMP_NUM_THREADS=1 # $SLURM_CPUS_PER_TASK
+ulimit -s unlimited
+sed -i '40 i sed -i s/"O4"/"O3"/g Makefile' $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_this_example.sh
+time ./run_this_example.sh
+grep GPU $install_dir/specfem3d_globe/DATA/Par_file
+
+echo "=========="
+echo "config.log"
+echo "=========="
+cat $install_dir/specfem3d_globe/config.log
+echo "========"
+echo "make.log"
+echo "========"
+cat $install_dir/specfem3d_globe/make.log
+
+echo "running seismogram comparisons:"
+echo
+
+cd $install_dir/specfem3d_globe/
+# uncompress seismograms
+if [ -e EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES_reference_OK/II.AAK.MXE.sem.ascii.bz2 ]; then
+ echo
+ echo "unzipping references..."
+ echo
+ mkdir OUTPUT_FILES_reference_OK/
+ bunzip2 EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES_reference_OK/*.bz2
+ echo
+ echo
+fi
+
+# compares seismograms by plotting correlations
+# Python2 (2.7 -virtual env - numpy)
+source /m100/home/userexternal/cjourdai/numpy-test/bin/activate
+which python
+./utils/compare_seismogram_correlations.py EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/ EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES_reference_OK/
+
+echo
+echo "done"
+echo
+echo "========"
+cat $install_dir//specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/output_solver.txt
+echo "========"
+cat $install_dir//specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/output_mesher.txt
diff --git a/specfem3d/job_script/job_marconi100_test_case_A.slurmc b/specfem3d/job_script/job_marconi100_test_case_A.slurmc
new file mode 100644
index 0000000000000000000000000000000000000000..4390ceb564efc8e38df5de8cb74e470d341582f7
--- /dev/null
+++ b/specfem3d/job_script/job_marconi100_test_case_A.slurmc
@@ -0,0 +1,81 @@
+#!/bin/bash
+#SBATCH -J Test_case_A
+#SBATCH -A Ppp4x_5850
+#SBATCH -p m100_usr_prod
+#SBATCH --time 01:59:00
+#SBATCH --nodes=24
+#SBATCH --ntasks-per-node=4
+#SBATCH --cpus-per-task=8
+#SBATCH --output=specfem3D_%x_marconi100-xl-spectrumpi-GPU-%j.output
+#SBATCH --gres=gpu:4
+#SBATCH --gpus-per-node=4
+#SBATCH --hint=nomultithread
+#SBATCH --exclusive
+#SBATCH --qos=m100_qos_bprod
+
+source ../env/env_marconi100
+echo "Environment used:"
+echo "================="
+grep -E -v '^(#|$)' ../env/env_marconi100
+cat job_marconi100_test_case_A.slurm
+cd $install_dir/TestCaseA/specfem3d_globe
+grep GPU DATA/Par_file
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+
+ulimit -s unlimited
+MESHER_EXE=./bin/xmeshfem3D
+SOLVER_EXE=./bin/xspecfem3D
+
+echo "=========="
+echo "config.log"
+echo "=========="
+cat $install_dir/TestCaseA/specfem3d_globe/config.log
+echo "========"
+echo "make.log"
+echo "========"
+cat $install_dir/TestCaseA/specfem3d_globe/make.log
+
+# backup files used for this simulation
+cp DATA/Par_file OUTPUT_FILES/
+cp DATA/STATIONS OUTPUT_FILES/
+cp DATA/CMTSOLUTION OUTPUT_FILES/
+
+##
+## mesh generation
+##
+sleep 2
+
+echo
+echo `date`
+echo "starting MPI mesher"
+echo
+
+MPI_PROCESS=$SLURM_NTASKS
+echo "SLURM_NTASKS= " $SLURM_NTASKS
+echo "SLURM_NTASKS_PER_NODE = " $SLURM_NTASKS_PER_NODE
+echo "SLURM_CPUS_PER_TASKS = " $SLURM_CPUS_PER_TASK
+echo "SLURM_NNODES=" $SLURM_NNODES
+echo "MPI_PROCESS $MPI_PROCESS"
+
+time mpirun -gpu -np ${MPI_PROCESS} ${MESHER_EXE}
+# set the value of --ntasks-per-node to the number of MPI processes you want to run per node, and --cpus-per-task = OMP_NUM_THREADS (if you want to exploit the SMT in terms of number of OMP threads) or to 128 / (ntasks-per-node) (if you want to exploit the SMT in terms of number of MPI processes).
+echo " mesher done: `date`"
+echo
+
+##
+## forward simulation
+##
+sleep 2
+
+echo
+echo `date`
+echo starting run in current directory $PWD
+echo
+time mpirun -gpu -np ${MPI_PROCESS} ${SOLVER_EXE}
+
+cat $install_dir/TestCaseA/specfem3d_globe/OUTPUT_FILES/output_mesher.txt
+echo "========"
+cat $install_dir/TestCaseA/specfem3d_globe/OUTPUT_FILES/output_solver.txt
+
+echo "finished successfully"
+echo `date`
diff --git a/specfem3d/job_script/job_marconi100_test_case_B.slurm b/specfem3d/job_script/job_marconi100_test_case_B.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..f27d168c8965580cda6898bca98babf4d6bea1a1
--- /dev/null
+++ b/specfem3d/job_script/job_marconi100_test_case_B.slurm
@@ -0,0 +1,81 @@
+#!/bin/bash
+#SBATCH -J Test_case_B
+#SBATCH -A Ppp4x_5850
+#SBATCH -p m100_usr_prod
+#SBATCH --time 01:59:00
+#SBATCH --nodes=384
+#SBATCH --ntasks-per-node=4
+#SBATCH --cpus-per-task=8
+#SBATCH --output=specfem3D_%x_marconi100-xl-spectrumpi-GPU-%j.output
+#SBATCH --gres=gpu:4
+#SBATCH --gpus-per-node=4
+#SBATCH --hint=nomultithread
+#SBATCH --exclusive
+
+source ../env/env_marconi100
+echo "Environment used:"
+echo "================="
+grep -E -v '^(#|$)' ../env/env_marconi100
+cat job_marconi100_test_case_B.slurm
+cd $install_dir/TestCaseB/specfem3d_globe
+grep GPU DATA/Par_file
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+
+ulimit -s unlimited
+MESHER_EXE=./bin/xmeshfem3D
+SOLVER_EXE=./bin/xspecfem3D
+
+echo "=========="
+echo "config.log"
+echo "=========="
+cat $install_dir/TestCaseB/specfem3d_globe/config.log
+echo "========"
+echo "make.log"
+echo "========"
+cat $install_dir/TestCaseB/specfem3d_globe/make.log
+
+# backup files used for this simulation
+cp DATA/Par_file OUTPUT_FILES/
+cp DATA/STATIONS OUTPUT_FILES/
+cp DATA/CMTSOLUTION OUTPUT_FILES/
+
+##
+## mesh generation
+##
+sleep 2
+
+echo
+echo `date`
+echo "starting MPI mesher"
+echo
+
+MPI_PROCESS=` echo "$SLURM_NNODES*$SLURM_NTASKS_PER_NODE" | bc -l`
+echo "SLURM_NTASKS= " $SLURM_NTASKS
+echo "SLURM_NTASKS_PER_NODE = " $SLURM_NTASKS_PER_NODE
+echo "SLURM_CPUS_PER_TASKS = " $SLURM_CPUS_PER_TASK
+echo "SLURM_NNODES=" $SLURM_NNODES
+echo "MPI_PROCESS $MPI_PROCESS"
+
+time mpirun -gpu -np ${MPI_PROCESS} ${MESHER_EXE}
+# set the value of --ntasks-per-node to the number of MPI processes you want to run per node, and --cpus-per-task = OMP_NUM_THREADS (if you want to exploit the SMT in terms of number of OMP threads) or to 128 / (ntasks-per-node) (if you want to exploit the SMT in terms of number of MPI processes).
+echo " mesher done: `date`"
+echo
+
+##
+## forward simulation
+##
+sleep 2
+
+echo
+echo `date`
+echo starting run in current directory $PWD
+echo
+time mpirun -gpu -np ${MPI_PROCESS} ${SOLVER_EXE}
+
+ls -lrth $install_dir/TestCaseB/specfem3d_globe/OUTPUT_FILES/output_*.txt
+cat $install_dir/TestCaseB/specfem3d_globe/OUTPUT_FILES/output_solver.txt
+echo "========"
+cat $install_dir/TestCaseB/specfem3d_globe/OUTPUT_FILES/output_mesher.txt
+
+echo "finished successfully"
+echo `date`
diff --git a/specfem3d/job_script/job_strong_scaling_vega-gpu_small_benchmark_run_to_test_more_complex_Earth.slurm b/specfem3d/job_script/job_strong_scaling_vega-gpu_small_benchmark_run_to_test_more_complex_Earth.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..af3009e63b8c4145f4cfa80cb4b7a6cc3a1ce620
--- /dev/null
+++ b/specfem3d/job_script/job_strong_scaling_vega-gpu_small_benchmark_run_to_test_more_complex_Earth.slurm
@@ -0,0 +1,49 @@
+#!/bin/bash
+#SBATCH -J specfem_strong_scaling_gpu_small_benchmark_run_to_test_more_complex_Earth
+#SBATCH --nodes=24
+#SBATCH --ntasks-per-node=1
+#SBATCH --cpus-per-task=128
+#SBATCH --time=00:59:59
+#SBATCH --output specfem_gpu_small_benchmark_run_to_test_more_complex_Earth-24Nodes-%j.out
+#SBATCH --exclusive
+#SBATCH -p gpu
+#SBATCH --gres=gpu:4
+#set -e
+source ../env/env_vega-gpu
+grep "^[^#;]" ../env/env_vega-gpu
+
+cat job_strong_scaling_vega-gpu_small_benchmark_run_to_test_more_complex_Earth.slurm
+
+cd $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth
+export OMPI_MCA_pml=ucx
+export OMPI_MCA_btl="^uct,tcp,openib,vader"
+export CUDA_VISIBLE_DEVICES=0,1,2,3
+#export OMP_NUM_THREADS=1
+
+sed -i s/"GPU_MODE = .false."/"GPU_MODE = .true."/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/DATA/Par_file
+sed -i s/"GPU_DEVICE = Tesla"/"GPU_DEVICE = *"/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/DATA/Par_file
+sed -i s/"configure --enable-openmp"/"configure --build=ppc64 --with-cuda=cuda8 "/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_this_example.sh
+#sed -i s/"mpirun -np"/"srun -n"/g $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/run_mesher_solver.bash
+#taskset -a -p $PPID
+
+time ./run_this_example.sh
+
+echo "=========="
+echo "config.log"
+echo "=========="
+cat $install_dir/specfem3d_globe/config.log
+echo "========"
+echo "make.log"
+echo "========"
+cat $install_dir/specfem3d_globe/make.log
+
+echo
+echo "running seismogram comparisons:"
+echo
+
+cd $install_dir/specfem3d_globe/
+echo "================================="
+echo "done"
+ls -lrth $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/output_*.txt
+cat $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/output_mesher.txt
+cat $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/output_solver.txt
diff --git a/specfem3d/job_script/job_supermuc-ng_small_benchmark_run_to_test_more_complex_Earth.slurm b/specfem3d/job_script/job_supermuc-ng_small_benchmark_run_to_test_more_complex_Earth.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..2ed94d40fbf82afc08b4131000146a2b34ab04b5
--- /dev/null
+++ b/specfem3d/job_script/job_supermuc-ng_small_benchmark_run_to_test_more_complex_Earth.slurm
@@ -0,0 +1,51 @@
+#!/bin/bash
+#SBATCH -J Validation_case_specfem-small_benchmark_run_to_test_more_complex_Earth
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=24
+#SBATCH --time=01:59:59
+#SBATCH --no-requeue
+#SBATCH --account=pn68go
+#SBATCH --partition=micro # insert test, micro, general, large or fat
+#SBATCH --output Validation_case_specfem_small_benchmark_run_to_test_more_complex_Earth-%j.out
+
+source ../env/env_supermuc-ng
+echo "Environment used:"
+echo "================="
+cat ../env/env_supermuc-ng
+cat job_supermuc-ng_small_benchmark_run_to_test_more_complex_Earth.slurm
+
+cd $install_dir/specfem3d_globe/EXAMPLES/small_benchmark_run_to_test_more_complex_Earth
+
+export SLURM_CPU_BIND=NONE
+export I_MPI_PIN=1
+export LIBRARY_PATH=$LD_LIBRARY_PATH
+echo "LD_LIBRARY_PATH = $LD_LIBRARY_PATH"
+export CPATH=$CPATH:/usr/local/include/:/usr/include
+export FPATH=$FPATH:/usr/local/include/:/usr/include
+
+echo "==============================================================="
+
+time ./run_mesher_solver.bash
+echo
+echo "running seismogram comparisons:"
+echo
+
+cd $install_dir/specfem3d_globe/
+# uncompress seismograms
+if [ -e EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES_reference_OK/II.AAK.MXE.sem.ascii.bz2 ]; then
+ echo
+ echo "unzipping references..."
+ echo
+ mkdir OUTPUT_FILES_reference_OK/
+ bunzip2 EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES_reference_OK/*.bz2
+ echo
+ echo
+fi
+
+module load python/2.7_intel
+# compares seismograms by plotting correlations
+./utils/compare_seismogram_correlations.py EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES/ EXAMPLES/small_benchmark_run_to_test_more_complex_Earth/OUTPUT_FILES_reference_OK/
+
+echo
+echo "done"
+echo
diff --git a/specfem3d/job_script/job_supermuc-ng_test_case_A.slurm b/specfem3d/job_script/job_supermuc-ng_test_case_A.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..85f5a5141a6ac0afa3c313b8719e129361f97884
--- /dev/null
+++ b/specfem3d/job_script/job_supermuc-ng_test_case_A.slurm
@@ -0,0 +1,89 @@
+#!/bin/bash
+#SBATCH -J Test_case_A
+#SBATCH --nodes=12
+#SBATCH --ntasks-per-node=8
+#SBATCH --cpus-per-task=6
+#SBATCH --time=01:00:00
+#SBATCH --no-requeue
+#SBATCH --account=pn73ve
+#SBATCH --partition=micro #general # insert test, micro, general, large or fat
+#SBATCH -o ./%x-12Nodes-6OMP-%j.out
+#set -e
+source ../env/env_supermuc-ng
+cat ../env/env_supermuc-ng
+grep "^[^#;]" ../env/env_supermuc-ng
+cat job_supermuc-ng_test_case_A.slurm
+echo "=========="
+echo "config.log"
+echo "=========="
+cat $install_dir/TestCaseA/specfem3d_globe/config.log
+echo "========"
+echo "make.log"
+echo "========"
+cat $install_dir/TestCaseA/specfem3d_globe/make.log
+echo "========"
+echo "Par_file"
+echo "========"
+cat $install_dir/TestCaseA/specfem3d_globe/DATA/Par_file
+
+module load slurm_setup
+
+cd $install_dir/TestCaseA/specfem3d_globe
+
+#Make sure that OMP_NUM_THREADS / KMP_HW_SUBSET = cpus-per-task
+export KMP_HW_SUBSET=1T
+export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
+#export KMP_AFFINITY=granularity=core,compact
+export FORT_BUFFERED=true
+export FORT_BLOCKSIZE=16777216
+
+ulimit -s unlimited
+
+export LIBRARY_PATH=$LD_LIBRARY_PATH
+echo "LD_LIBRARY_PATH = $LD_LIBRARY_PATH"
+
+MESHER_EXE=./bin/xmeshfem3D
+SOLVER_EXE=./bin/xspecfem3D
+
+# backup files used for this simulation
+cp DATA/Par_file OUTPUT_FILES/
+cp DATA/STATIONS OUTPUT_FILES/
+cp DATA/CMTSOLUTION OUTPUT_FILES/
+##
+## mesh generation
+##
+sleep 2
+
+echo
+echo `date`
+echo "starting MPI mesher"
+echo
+
+MPI_PROCESS=` echo "$SLURM_NNODES*$SLURM_NTASKS_PER_NODE" | bc -l`
+echo "SLURM_NTASKS_PER_NODE = " $SLURM_NTASKS_PER_NODE
+echo "SLURM_CPUS_PER_TASKS = " $SLURM_CPUS_PER_TASK
+echo "SLURM_NNODES=" $SLURM_NNODES
+echo "MPI_PROCESS $MPI_PROCESS"
+echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
+
+time mpiexec -np ${MPI_PROCESS} ${MESHER_EXE}
+echo " mesher done: `date`"
+echo
+
+##
+## forward simulation
+##
+sleep 2
+
+echo
+echo `date`
+echo starting run in current directory $PWD
+echo
+time mpiexec -np ${MPI_PROCESS} ${SOLVER_EXE}
+
+echo "finished successfully"
+echo "====================="
+echo `date`
+ls -lrth $install_dir/TestCaseA/specfem3d_globe/OUTPUT_FILES/output_*.txt
+cat $install_dir/TestCaseA/specfem3d_globe/OUTPUT_FILES/output_mesher.txt
+cat $install_dir/TestCaseA/specfem3d_globe/OUTPUT_FILES/output_solver.txt
diff --git a/specfem3d/job_script/job_supermuc-ng_test_case_B.slurm b/specfem3d/job_script/job_supermuc-ng_test_case_B.slurm
new file mode 100644
index 0000000000000000000000000000000000000000..6a36b994a6f4378b1f8f5286e5771dcac4bb4f43
--- /dev/null
+++ b/specfem3d/job_script/job_supermuc-ng_test_case_B.slurm
@@ -0,0 +1,72 @@
+#!/bin/bash
+#SBATCH -J Test_case_B
+#SBATCH --nodes=384
+#SBATCH --ntasks-per-node=4
+#SBATCH --cpus-per-task=12
+#SBATCH --time=00:29:59
+#SBATCH --no-requeue
+#SBATCH --account=pn68go
+#SBATCH --partition=general # insert test, micro, general, large or fat
+#SBATCH -o ./%x-12OMP.%j.out
+#set -e
+source ../env/env_supermuc-ng
+cat ../env/env_supermuc-ng
+cat job_supermuc-ng_test_case_A.slurm
+module load slurm_setup
+
+cd $install_dir/TestCaseB/specfem3d_globe
+
+#Make sure that OMP_NUM_THREADS / KMP_HW_SUBSET = cpus-per-task
+export KMP_HW_SUBSET=1T
+export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
+#export KMP_AFFINITY=granularity=core,compact
+export FORT_BUFFERED=true
+export FORT_BLOCKSIZE=16777216
+
+ulimit -s unlimited
+
+export LIBRARY_PATH=$LD_LIBRARY_PATH
+echo "LD_LIBRARY_PATH = $LD_LIBRARY_PATH"
+
+MESHER_EXE=./bin/xmeshfem3D
+SOLVER_EXE=./bin/xspecfem3D
+
+# backup files used for this simulation
+cp DATA/Par_file OUTPUT_FILES/
+cp DATA/STATIONS OUTPUT_FILES/
+cp DATA/CMTSOLUTION OUTPUT_FILES/
+
+##
+## mesh generation
+##
+sleep 2
+
+echo
+echo `date`
+echo "starting MPI mesher"
+echo
+
+MPI_PROCESS=` echo "$SLURM_NNODES*$SLURM_NTASKS_PER_NODE" | bc -l`
+echo "SLURM_NTASKS_PER_NODE = " $SLURM_NTASKS_PER_NODE
+echo "SLURM_CPUS_PER_TASKS = " $SLURM_CPUS_PER_TASK
+echo "SLURM_NNODES=" $SLURM_NNODES
+echo "MPI_PROCESS $MPI_PROCESS"
+echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
+
+time mpiexec -np ${MPI_PROCESS} ${MESHER_EXE}
+echo " mesher done: `date`"
+echo
+
+##
+## forward simulation
+##
+sleep 2
+
+echo
+echo `date`
+echo starting run in current directory $PWD
+echo
+time mpiexec -np ${MPI_PROCESS} ${SOLVER_EXE}
+
+echo "finished successfully"
+echo `date`