gpaw/README.md

@@ -100,7 +100,7 @@ these versions of GPAW.
 Installing and running GPAW has changed a lot in the since the previous
 versions of the UEABS. GPAW version numbering changed in 2019. Version 1.5.3 is the
 last version with the old numbering. In 2019 the development team switched
-to a version numbering scheme based on year, month and patchlevel, e.g.,
+to a version numbering scheme based on year, month and patch level, e.g.,
 19.8.1 for the second version released in August 2019.
 
 Another change is in the Python packages used to install GPAW. Versions up to
@@ -169,9 +169,9 @@ In addition, the GPU version needs:
 
 Installing GPAW also requires a number of standard build tools on the system, including
   * [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
-    configure script for libxc
+    configure script for LibXC
   * [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
-    the configure process of libxc produces very misleading
+    the configure process of LibXC produces very misleading
     error messages that do not immediately point to libtool missing.
   * [GNU make](https://www.gnu.org/software/make/)
 
@@ -297,8 +297,8 @@ on one hand and version 21.1.0 on the other hand.
 
 The expected values are:
   * Number of iterations: Between 30 and 35
-  * Dipole (3rd component):between -0.493 and -0.491
-  * Fermi level:between -2.67 and -2.66
+  * Dipole (3rd component): Between -0.493 and -0.491
+  * Fermi level: Between -2.67 and -2.66
   * Extrapolated energy: Between -3784 and -3783
 
 Note: Though not used for the benchmarking in the final report, some testing was done
@@ -307,7 +307,7 @@ by new internal implementations that cause changes in some results. For 21.1.0,
 expected values are:
 
   * Number of iterations: Between 30 and 35
-  * Dipole (3rd component): between -0.462 and -0.461
-  * Fermi level: between -2.59 and -2.58
+  * Dipole (3rd component): Between -0.462 and -0.461
+  * Fermi level: Between -2.59 and -2.58
   * Extrapolated energy: Between -3784 and -3783
 

gpaw/build/build-CPU.md

-# Detailed GPAW installation instructions on non-acclerated systems
+# Detailed GPAW installation instructions on non-accelerated systems
 
 These instructions are in addition to the brief instructions in [README.md](../README.md).
 
@@ -13,7 +13,7 @@ GPAW needs (for the UEABS benchmarks)
   * [Python](https://www.python.org/): GPAW 20.1.0 requires Python 3.5-3.8, and
     GPAW 20.10.0 and 21.1.0 require Python 3.6-3.9.
   * [MPI library](https://www.mpi-forum.org/)
-  * [LibXC](https://www.tddft.org/programs/libxc/). GPAW 20.1.0,
+  * [LibXC](https://www.tddft.org/programs/LibXC/). GPAW 20.1.0,
     20.10.0 and 21.1.0 all need LibXC 3.x or 4.x.
   * (Optimized) [BLAS](http://www.netlib.org/blas/) and
     [LAPACK](http://www.netlib.org/lapack/) libraries.
@@ -22,7 +22,7 @@ GPAW needs (for the UEABS benchmarks)
     will give very poor performance. Most optimized LAPACK libraries actually only
     optimize a few critical routines while the remaining routines are compiled from
     the reference version. Most processor vendors for HPC machines and system vendors
-    offer optmized versions of these libraries.
+    offer optimized versions of these libraries.
   * [ScaLAPACK](http://www.netlib.org/scalapack/) and the underlying communication
     layer [BLACS](http://www.netlib.org/blacs/).
   * [FFTW](http://www.fftw.org/) or compatible FFT library.
@@ -60,7 +60,7 @@ GPAW needs
   * [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
     from the same group that develops GPAW. The required versions is 3.18.0 or later for
     GPAW 20.1.0, 20.10.0 and 21.1.0.
-    ASE has a couple of dependendencies
+    ASE has a couple of dependencies
     that are not needed for running the UEABS benchmarks. However, several Python
     package install methods will trigger the installation of those packages, and
     with them may require a chain of system libraries.
@@ -69,7 +69,7 @@ GPAW needs
         This package is optional and not really needed to run the benchmarks.
         Matplotlib pulls in a lot of other dependencies. When installing ASE with pip,
         it will try to pull in matplotlib and its dependencies
-          * [pillow](https://pypi.org/project/Pillow/) needs several exgternal
+          * [pillow](https://pypi.org/project/Pillow/) needs several external
             libraries. During the development of the benchmarks, we needed at least
             zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even
             though the pillow documentation claimed that libjpeg was optional,
@@ -90,7 +90,7 @@ GPAW needs
                 code
           * [itsdangerous](https://pypi.org/project/itsdangerous/)
           * [Werkzeug](https://pypi.org/project/Werkzeug/)
-          * [click]()
+          * [click](https://pypi.org/project/click/)
 
 
 ## Tested configurations
@@ -145,28 +145,28 @@ Also, the instructions below will need to be adapted to the specific
 libraries that are being used.
 
 Other prerequisites:
-  * libxc
+  * LibXC
   * Python interpreter
   * Python package NumPy
   * Python package SciPy
   * Python package ase
 
 
-### Installing libxc
+### Installing LibXC
 
-  * Installing libxc requires GNU automake and GNU buildtool besides GNU make and a
+  * Installing LibXC requires GNU automake and GNU buildtool besides GNU make and a
     C compiler. The build process is the usual GNU configure - make - make install
     cycle, but the `configure` script still needs to be generated with autoreconf.
-  * Download libxc:
-      * The latest version of libxc can be downloaded from
-        [the libxc download page](https://www.tddft.org/programs/libxc/download/).
+  * Download LibXC:
+      * The latest version of LibXC can be downloaded from
+        [the LibXC download page](https://www.tddft.org/programs/libxc/download/).
         However, that version may not be officially supported by GPAW.
-      * It is also possible to download all recent versions of libxc from
-        [the libxc GitLab](https://gitlab.com/libxc/libxc)
+      * It is also possible to download all recent versions of LibXC from
+        [the LibXC GitLab](https://gitlab.com/libxc/libxc)
           * Select the tag corresponding to the version you want to download in the
             branch/tag selection box.
           * Then use the download button and select the desired file type.
-          * Dowload URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
+          * Download URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
   * Untar the file in the build directory.
 
 
@@ -228,8 +228,8 @@ of NumPy, SciPy and GPAW itself proves much more important.
     use the NumPy FFT routines.
   * GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files
     can be found on the [GPAW website, Atomic PAW Setups page](https://wiki.fysik.dtu.dk/gpaw/setups/setups.html).
-    For the testing we used []`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
-    for all versions of GPAW. The easiest way to install these files is to simpy untar
+    For the testing we used [`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
+    for all versions of GPAW. The easiest way to install these files is to simply untar
     the file and set the environment variable GPAW_SETUP_PATH to point to that directory.
     In the examples provided we use the `share/gpaw-setups` subdirectory of the install
     directory for this purpose.

gpaw/build/examples/LRZ-SuperMUCng-skylake/README.md

+# Build issues at LRZ
+
+## Compilers used for testing
+
+The following compilers, available on the system at the time of testing and installed
+with Spack, were used:
+
+  * Intel C/C++/Fortran version 19.1.2.254 (as provided by the intel/2020.0.2 module)
+  * Intel MPI version 2019.8.254
+  * Intel MKL 2020.1.217
+
+These are components from the Intel compilers version 2020 update 2.
+
+## Issues
+
+  * It is impossible to download files from this system, so all sources have to be
+    put in place manually before starting the compile. The scripts will generate the
+    necessary URLs.
+
+  * A general issues: At some patch releases of Python 3.8 and 3.9 changes were made
+    to certain components of the code that installs additional packages. This causes
+    failures in the installation process of NumPy and SciPy. So it is better to stick
+    to the versions used in these example scripts (Python 3.8.7 and 3.9.4).
+
+  * GPAW tried to compile a file that includes an MPI header with the regular C compiler.
+    The solution was to set the regular C compiler to the MPI wrapper compiler script.
+
+    That issue does not occur on all systems since some systems have the MPI header
+    files included in ``CFILES`` so that the regular C compile also finds those header
+    files.
+
+  * We failed to get the combination Python 3.9.4 + NumPy 1.19.5 + SciPy 1.5.4 to compile.
+    It failed during the installation of SciPy, even though this combination worked on
+    several other systems.
+
+  * The system uses TCL modules 4.6 at the time of testing. We failed to get the right
+    compiler (which were not the default on the system) to load in the module file
+    unless we first manually unloaded intel-mkl, intel-mpi and intel.
+
+    There are no examples on how to do this in the existing module files of the system
+    which may be due to the fact that spack with rpath is being used.
+
+    It might have been a better solution to simply unload spack and use the other
+    compiler modules that are not spack-managed instead rather than a variant of the
+    default environment as presented by the system.

gpaw/build/examples/README.md

 # Example build scripts
 
+## BSC-MareNostrum4-skylake
+
+MareNostrum 4 is a Tier-0 cluster installed at the Barcelona Supercomputing
+Center. GPAW was installed on the general-purpose block of the cluster,
+consisting of 3,456 nodes with dual 24-core Intel Xeon Platinum 8160
+"Skylake" CPUs and 96 GB of RAM in most nodes. The interconnect is
+Omni-Path.
+
+Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
+
+[Go to the directory](BSC-MareNostrum4-skylake)
+
+
+## HLRS-Hawk-rome
+
+Hawk is a Tier-0 cluster hosted by HLRS in Germany. The cluster consists of
+5,632 compute nodes with dual 64-core AMD EPYC 7742 "Rome" CPUs and 256 GB of
+memory. The interconnect is HDR200 InfiniBand in a 9-dimensional hypercube topology.
+
+Most software on the cluster is installed through [Spack](https://spack.io/).
+Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
+
+[Go to the directory](HLRS-Hawk-rome)
+
+
+## JSC-JUWELS-skylake
+
+JUWELS is a Tier-0 system hosted by the Jülich Supercomputing Centre. GPAW was only
+installed on the so-called "Cluster Module" consisting of 2,271 compute nodes
+with dual 24-core Intel Xeon Platinum 8168 "Skylake" CPUs and 96 GB RAM per node.
+The interconnect is EDR InfiniBand.
+
+Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
+Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
+
+[Go to the directory](JSC-JUWELS-skylake)
+
+
+## LRZ-SuperMUCng-skylake
+
+SuperMUC-NG is a Tier-0 system hosted by LRZ in Germany. The cluster has 6,480
+compute nodes with dual 24-core Intel Xeon Platinum 8174 CPUs. Most nodes have 96 GB
+of RAM. The interconnect is Omni-Path. The cluster runs a SUSE Linux variant.
+
+Most software on the cluster is installed through [Spack](https://spack.io/).
+Environment modules are provided through the TCL-based [Environment Modules
+package](https://modules.readthedocs.io/).
+
+[Go to the directory](LRZ-SuperMUCng-skylake)
+
+
+## TGCC-Irene-rome
+
+Joliot-Curie/Irene is a Tier-0 machine hosted at the TGCC in France. GPAW was installed
+on the AMD Rome partition. This partition has 2,292 dual 64-core AMD EPYC 7H12 "Rome"
+CPUs with 256 GB RAM each. The interconnect is HDR100 InfiniBand.
+
+Environment modules are provided through the TCL-based [Environment Modules
+package](https://modules.readthedocs.io/).
+
+[Go to the directory](TGCC-Irene-rome)
+
+
 ## CalcUA-vaughan-rome
 
-Vaughan is a cluster installed at the University of Antwerpen
+Vaughan is a cluster installed at the University of Antwerp
 as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
-The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256GB of RAM memory
+The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256 GB of RAM memory
 per node. Nodes are interconnected through an InfiniBand HDR100 network. The system
 does not contain accelerators. The cluster uses CentOS 8 as the
 operating system.
 
-Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
+Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
 Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
 
 [Go to the directory](CalcUA-vaughan-rome)
@@ -17,50 +80,30 @@ Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/l
 
 ## CalcUA-leibniz-broadwell
 
-Leibniz is a cluster intalled at the University of Antwerpen
+Leibniz is a cluster installed at the University of Antwerp
 as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
 The cluster has 153 regular CPU nodes with dual Intel Xeon E5-2680v4
 "broadwell" CPUs with 128 or 256 GB of RAM memory per node.
-Nodes are interconnected through an Infiniband EDR network.
+Nodes are interconnected through an InfiniBand EDR network.
 The system has two nodes with dual NVIDIA P100 GPUs and a node
 with dual NEC Aurora TSUBASA first generation vector boards.
 During the testing period the cluster used CentOS 7 as the
 operating system.
 
-Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
+Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
 Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
 
 [Go to the directory](CalcUA-leibniz-broadwell)
 
 
-## davide @ Cineca
-
-Build instructions for GPU and CPU on the DAVIDE OpenPower GPU cluster @ Cineca.
-
-These instructions haven't been tested recently. They are still for a version
-based on distutils rather then setuptools that was dropped towards the end of
-the development of this version of the benchmark suite.
-
-[Go to the directory](davide)
-
-
-## juwels @ Jülich Supercomputing Centre
-
-Build instructions for the CPU nodes of JUWELS.
-
-These instructions haven't been tested recently. They are still for a version
-based on distutils rather then setuptools that was dropped towards the end of
-the development of this version of the benchmark suite.
-
-[Go to the directory](juwels)
-
-
 ## piz-daint @ CSCS
 
 Build instructions for the Piz Daint (GPU cluster).
 
 These instructions haven't been tested recently. They are still for a version
-based on distutils rather then setuptools that was dropped towards the end of
-the development of this version of the benchmark suite.
+based on distutils rather than setuptools that was dropped towards the end of
+the development of this version of the benchmark suite. They are here because
+it is the only still relevant set of instructions for a GPU-based machine.
 
 [Go to the directory](piz-daint)
+

gpaw/build/examples/davide/gpaw-cpu/build.sh

-### GPAW installation script for D.A.V.I.D.E
-
-# version numbers (modify if needed)
-gpaw_version=1.1.0
-
-# installation directory (modify!)
-tgt=$CINECA_SCRATCH/lib/gpaw-${gpaw_version}
-
-# setup build environment
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-source $CINECA_SCRATCH/lib/python-2018-12-cuda/load.sh 2016-06
-source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
-export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
-export CFLAGS=""
-
-# gpaw
-git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
-cd gpaw-$gpaw_version
-git checkout $gpaw_version
-patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
-cp ../setup/customize-power8.py .
-python setup.py install --customize=customize-power8.py --prefix=$tgt 2>&1 | tee loki-inst
-cd ..
-sed -e "s|<BASE>|$tgt|g" -e "s|<PYTHONHOME>|$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt

gpaw/build/examples/davide/gpaw-cpu/setup/customize-power8.py

-# Setup customisation for gpaw/cuda
-import os
-
-# compiler and linker
-compiler = 'gcc'
-mpicompiler = 'mpicc'
-mpilinker = 'mpicc'
-extra_compile_args = ['-std=c99', '-mcpu=power8']
-
-# libraries
-libraries = ['z']
-
-# openblas
-library_dirs += [os.environ['OPENBLAS_ROOT'] + '/lib']
-include_dirs += [os.environ['OPENBLAS_ROOT'] + '/include']
-libraries += ['openblas']
-
-# scalapack
-library_dirs += [os.environ['SCALAPACK_ROOT'] + '/lib']
-libraries += ['scalapack']
-
-# libxc
-library_dirs += [os.environ['LIBXCDIR'] + '/lib']
-include_dirs += [os.environ['LIBXCDIR'] + '/include']
-libraries += ['xc']
-
-# GPAW defines
-define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
-define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
-define_macros += [("GPAW_ASYNC",1)]
-define_macros += [("GPAW_MPI2",1)]
-define_macros += [('GPAW_CUDA', '1')]
-
-# ScaLAPACK
-scalapack = True
-
-# HDF5
-hdf5 = False
-

gpaw/build/examples/davide/gpaw-cpu/setup/load-gpaw.sh

-#!/bin/bash
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-source <PYTHONHOME>/load.sh
-source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
-export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
-export PATH=<BASE>/bin:$PATH
-export PYTHONPATH=<BASE>/lib/python2.7/site-packages:$PYTHONPATH

gpaw/build/examples/davide/gpaw/build.sh

-### GPAW installation script for D.A.V.I.D.E
-
-# version numbers (modify if needed)
-gpaw_version=cuda
-
-# installation directory (modify!)
-tgt=$CINECA_SCRATCH/lib/gpaw-${gpaw_version}
-
-# setup build environment
-module load cuda/9.2.88
-module load cudnn/7.1.4--cuda--9.2.88
-module load gnu/6.4.0
-module load openmpi/3.1.0--gnu--6.4.0
-source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
-source $CINECA_SCRATCH/lib/python-2018-12-cuda/load.sh 2016-06
-source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
-export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
-export CFLAGS=""
-
-# gpaw
-git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
-cd gpaw-$gpaw_version
-git checkout $gpaw_version
-patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
-cp ../setup/customize-cuda.py .
-cd c/cuda
-cp ../../../setup/make.inc .
-make 2>&1 | tee loki-make
-cd -
-python setup.py install --customize=customize-cuda.py --prefix=$tgt 2>&1 | tee loki-inst
-cd ..
-sed -e "s|<BASE>|$tgt|g" -e "s|<PYTHONHOME>|$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
-
-# fix permissions
-chmod -R g+rwX $tgt
-chmod -R o+rX $tgt