Skip to content
GitLab
Commit bd4b310b authored by Cedric Jourdain's avatar Cedric Jourdain 🐵
Browse files

Merge branch 'r2.2-dev' of https://repository.prace-ri.eu/git/UEABS/ueabs into r2.2-dev

parents 420d3ef6 f75ba65b
Expand all Hide whitespace changes
Inline Side-by-side
README.md
View file @ bd4b310b
......@@ -282,7 +282,7 @@ The SHOC benchmark suite currently contains benchmark programs, categoried based
# SPECFEM3D <a name="specfem3d"></a>
| **General information** | **Scientific field** | **Language** | **MPI** | **OpenMP** | **GPU** | **LoC** | **Code description** |
|------------------|----------------------|--------------|---------|------------|---------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| [- Website](https://geodynamics.org/cig/software/specfem3d_globe/) <br>[- Source](https://github.com/geodynamics/specfem3d_globe.git) <br>[- Bench](https://repository.prace-ri.eu/git/UEABS/ueabs/tree/r2.1-dev/specfem3d) <br>[- Summary](https://repository.prace-ri.eu/git/UEABS/ueabs/blob/r2.1-dev/specfem3d/PRACE_UEABS_Specfem3D_summary.pdf) | Geodynamics | Fortran | yes | yes | Yes (CUDA) | 140000 | The software package SPECFEM3D simulates three-dimensional global and regional seismic wave propagation based upon the spectral-element method (SEM). |
| [- Website](https://geodynamics.org/cig/software/specfem3d_globe/) <br>[- Source](https://github.com/geodynamics/specfem3d_globe.git) <br>[- Bench](https://repository.prace-ri.eu/git/UEABS/ueabs/tree/r2.1-dev/specfem3d) <br>[- Summary](https://repository.prace-ri.eu/git/UEABS/ueabs/blob/r2.1-dev/specfem3d/PRACE_UEABS_Specfem3D_summary.pdf) | Geodynamics | Fortran & C | yes | yes | Yes (CUDA) | 100k Fortran & 20k C | The software package SPECFEM3D simulates three-dimensional global and regional seismic wave propagation based upon the spectral-element method (SEM). |
# TensorFlow <a name="tensorflow"></a>
......
gpaw/README.md
View file @ bd4b310b
......@@ -100,7 +100,7 @@ these versions of GPAW.
Installing and running GPAW has changed a lot in the since the previous
versions of the UEABS. GPAW version numbering changed in 2019. Version 1.5.3 is the
last version with the old numbering. In 2019 the development team switched
to a version numbering scheme based on year, month and patchlevel, e.g.,
to a version numbering scheme based on year, month and patch level, e.g.,
19.8.1 for the second version released in August 2019.
Another change is in the Python packages used to install GPAW. Versions up to
......@@ -169,9 +169,9 @@ In addition, the GPU version needs:
Installing GPAW also requires a number of standard build tools on the system, including
* [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
configure script for libxc
configure script for LibXC
* [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
the configure process of libxc produces very misleading
the configure process of LibXC produces very misleading
error messages that do not immediately point to libtool missing.
* [GNU make](https://www.gnu.org/software/make/)
......@@ -297,8 +297,8 @@ on one hand and version 21.1.0 on the other hand.
The expected values are:
* Number of iterations: Between 30 and 35
* Dipole (3rd component):between -0.493 and -0.491
* Fermi level:between -2.67 and -2.66
* Dipole (3rd component): Between -0.493 and -0.491
* Fermi level: Between -2.67 and -2.66
* Extrapolated energy: Between -3784 and -3783
Note: Though not used for the benchmarking in the final report, some testing was done
......@@ -307,7 +307,7 @@ by new internal implementations that cause changes in some results. For 21.1.0,
expected values are:
* Number of iterations: Between 30 and 35
* Dipole (3rd component): between -0.462 and -0.461
* Fermi level: between -2.59 and -2.58
* Dipole (3rd component): Between -0.462 and -0.461
* Fermi level: Between -2.59 and -2.58
* Extrapolated energy: Between -3784 and -3783
gpaw/build/build-CPU.md
View file @ bd4b310b
# Detailed GPAW installation instructions on non-acclerated systems
# Detailed GPAW installation instructions on non-accelerated systems
These instructions are in addition to the brief instructions in [README.md](../README.md).
......@@ -13,7 +13,7 @@ GPAW needs (for the UEABS benchmarks)
* [Python](https://www.python.org/): GPAW 20.1.0 requires Python 3.5-3.8, and
GPAW 20.10.0 and 21.1.0 require Python 3.6-3.9.
* [MPI library](https://www.mpi-forum.org/)
* [LibXC](https://www.tddft.org/programs/libxc/). GPAW 20.1.0,
* [LibXC](https://www.tddft.org/programs/LibXC/). GPAW 20.1.0,
20.10.0 and 21.1.0 all need LibXC 3.x or 4.x.
* (Optimized) [BLAS](http://www.netlib.org/blas/) and
[LAPACK](http://www.netlib.org/lapack/) libraries.
......@@ -22,7 +22,7 @@ GPAW needs (for the UEABS benchmarks)
will give very poor performance. Most optimized LAPACK libraries actually only
optimize a few critical routines while the remaining routines are compiled from
the reference version. Most processor vendors for HPC machines and system vendors
offer optmized versions of these libraries.
offer optimized versions of these libraries.
* [ScaLAPACK](http://www.netlib.org/scalapack/) and the underlying communication
layer [BLACS](http://www.netlib.org/blacs/).
* [FFTW](http://www.fftw.org/) or compatible FFT library.
......@@ -60,7 +60,7 @@ GPAW needs
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW. The required versions is 3.18.0 or later for
GPAW 20.1.0, 20.10.0 and 21.1.0.
ASE has a couple of dependendencies
ASE has a couple of dependencies
that are not needed for running the UEABS benchmarks. However, several Python
package install methods will trigger the installation of those packages, and
with them may require a chain of system libraries.
......@@ -69,7 +69,7 @@ GPAW needs
This package is optional and not really needed to run the benchmarks.
Matplotlib pulls in a lot of other dependencies. When installing ASE with pip,
it will try to pull in matplotlib and its dependencies
* [pillow](https://pypi.org/project/Pillow/) needs several exgternal
* [pillow](https://pypi.org/project/Pillow/) needs several external
libraries. During the development of the benchmarks, we needed at least
zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even
though the pillow documentation claimed that libjpeg was optional,
......@@ -90,7 +90,7 @@ GPAW needs
code
* [itsdangerous](https://pypi.org/project/itsdangerous/)
* [Werkzeug](https://pypi.org/project/Werkzeug/)
* [click]()
* [click](https://pypi.org/project/click/)
## Tested configurations
......@@ -145,28 +145,28 @@ Also, the instructions below will need to be adapted to the specific
libraries that are being used.
Other prerequisites:
* libxc
* LibXC
* Python interpreter
* Python package NumPy
* Python package SciPy
* Python package ase
### Installing libxc
### Installing LibXC
* Installing libxc requires GNU automake and GNU buildtool besides GNU make and a
* Installing LibXC requires GNU automake and GNU buildtool besides GNU make and a
C compiler. The build process is the usual GNU configure - make - make install
cycle, but the `configure` script still needs to be generated with autoreconf.
* Download libxc:
* The latest version of libxc can be downloaded from
[the libxc download page](https://www.tddft.org/programs/libxc/download/).
* Download LibXC:
* The latest version of LibXC can be downloaded from
[the LibXC download page](https://www.tddft.org/programs/libxc/download/).
However, that version may not be officially supported by GPAW.
* It is also possible to download all recent versions of libxc from
[the libxc GitLab](https://gitlab.com/libxc/libxc)
* It is also possible to download all recent versions of LibXC from
[the LibXC GitLab](https://gitlab.com/libxc/libxc)
* Select the tag corresponding to the version you want to download in the
branch/tag selection box.
* Then use the download button and select the desired file type.
* Dowload URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
* Download URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
* Untar the file in the build directory.
......@@ -228,8 +228,8 @@ of NumPy, SciPy and GPAW itself proves much more important.
use the NumPy FFT routines.
* GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files
can be found on the [GPAW website, Atomic PAW Setups page](https://wiki.fysik.dtu.dk/gpaw/setups/setups.html).
For the testing we used []`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
for all versions of GPAW. The easiest way to install these files is to simpy untar
For the testing we used [`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
for all versions of GPAW. The easiest way to install these files is to simply untar
the file and set the environment variable GPAW_SETUP_PATH to point to that directory.
In the examples provided we use the `share/gpaw-setups` subdirectory of the install
directory for this purpose.
......
gpaw/build/examples/LRZ-SuperMUCng-skylake/README.md 0 → 100644
View file @ bd4b310b
# Build issues at LRZ
## Compilers used for testing
The following compilers, available on the system at the time of testing and installed
with Spack, were used:
* Intel C/C++/Fortran version 19.1.2.254 (as provided by the intel/2020.0.2 module)
* Intel MPI version 2019.8.254
* Intel MKL 2020.1.217
These are components from the Intel compilers version 2020 update 2.
## Issues
* It is impossible to download files from this system, so all sources have to be
put in place manually before starting the compile. The scripts will generate the
necessary URLs.
* A general issues: At some patch releases of Python 3.8 and 3.9 changes were made
to certain components of the code that installs additional packages. This causes
failures in the installation process of NumPy and SciPy. So it is better to stick
to the versions used in these example scripts (Python 3.8.7 and 3.9.4).
* GPAW tried to compile a file that includes an MPI header with the regular C compiler.
The solution was to set the regular C compiler to the MPI wrapper compiler script.
That issue does not occur on all systems since some systems have the MPI header
files included in ``CFILES`` so that the regular C compile also finds those header
files.
* We failed to get the combination Python 3.9.4 + NumPy 1.19.5 + SciPy 1.5.4 to compile.
It failed during the installation of SciPy, even though this combination worked on
several other systems.
* The system uses TCL modules 4.6 at the time of testing. We failed to get the right
compiler (which were not the default on the system) to load in the module file
unless we first manually unloaded intel-mkl, intel-mpi and intel.
There are no examples on how to do this in the existing module files of the system
which may be due to the fact that spack with rpath is being used.
It might have been a better solution to simply unload spack and use the other
compiler modules that are not spack-managed instead rather than a variant of the
default environment as presented by the system.
gpaw/build/examples/README.md
View file @ bd4b310b
# Example build scripts
## BSC-MareNostrum4-skylake
MareNostrum 4 is a Tier-0 cluster installed at the Barcelona Supercomputing
Center. GPAW was installed on the general-purpose block of the cluster,
consisting of 3,456 nodes with dual 24-core Intel Xeon Platinum 8160
"Skylake" CPUs and 96 GB of RAM in most nodes. The interconnect is
Omni-Path.
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](BSC-MareNostrum4-skylake)
## HLRS-Hawk-rome
Hawk is a Tier-0 cluster hosted by HLRS in Germany. The cluster consists of
5,632 compute nodes with dual 64-core AMD EPYC 7742 "Rome" CPUs and 256 GB of
memory. The interconnect is HDR200 InfiniBand in a 9-dimensional hypercube topology.
Most software on the cluster is installed through [Spack](https://spack.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](HLRS-Hawk-rome)
## JSC-JUWELS-skylake
JUWELS is a Tier-0 system hosted by the Jülich Supercomputing Centre. GPAW was only
installed on the so-called "Cluster Module" consisting of 2,271 compute nodes
with dual 24-core Intel Xeon Platinum 8168 "Skylake" CPUs and 96 GB RAM per node.
The interconnect is EDR InfiniBand.
Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](JSC-JUWELS-skylake)
## LRZ-SuperMUCng-skylake
SuperMUC-NG is a Tier-0 system hosted by LRZ in Germany. The cluster has 6,480
compute nodes with dual 24-core Intel Xeon Platinum 8174 CPUs. Most nodes have 96 GB
of RAM. The interconnect is Omni-Path. The cluster runs a SUSE Linux variant.
Most software on the cluster is installed through [Spack](https://spack.io/).
Environment modules are provided through the TCL-based [Environment Modules
package](https://modules.readthedocs.io/).
[Go to the directory](LRZ-SuperMUCng-skylake)
## TGCC-Irene-rome
Joliot-Curie/Irene is a Tier-0 machine hosted at the TGCC in France. GPAW was installed
on the AMD Rome partition. This partition has 2,292 dual 64-core AMD EPYC 7H12 "Rome"
CPUs with 256 GB RAM each. The interconnect is HDR100 InfiniBand.
Environment modules are provided through the TCL-based [Environment Modules
package](https://modules.readthedocs.io/).
[Go to the directory](TGCC-Irene-rome)
## CalcUA-vaughan-rome
Vaughan is a cluster installed at the University of Antwerpen
Vaughan is a cluster installed at the University of Antwerp
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256GB of RAM memory
The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256 GB of RAM memory
per node. Nodes are interconnected through an InfiniBand HDR100 network. The system
does not contain accelerators. The cluster uses CentOS 8 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](CalcUA-vaughan-rome)
......@@ -17,50 +80,30 @@ Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/l
## CalcUA-leibniz-broadwell
Leibniz is a cluster intalled at the University of Antwerpen
Leibniz is a cluster installed at the University of Antwerp
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 153 regular CPU nodes with dual Intel Xeon E5-2680v4
"broadwell" CPUs with 128 or 256 GB of RAM memory per node.
Nodes are interconnected through an Infiniband EDR network.
Nodes are interconnected through an InfiniBand EDR network.
The system has two nodes with dual NVIDIA P100 GPUs and a node
with dual NEC Aurora TSUBASA first generation vector boards.
During the testing period the cluster used CentOS 7 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](CalcUA-leibniz-broadwell)
## davide @ Cineca
Build instructions for GPU and CPU on the DAVIDE OpenPower GPU cluster @ Cineca.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
[Go to the directory](davide)
## juwels @ Jülich Supercomputing Centre
Build instructions for the CPU nodes of JUWELS.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
[Go to the directory](juwels)
## piz-daint @ CSCS
Build instructions for the Piz Daint (GPU cluster).
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
based on distutils rather than setuptools that was dropped towards the end of
the development of this version of the benchmark suite. They are here because
it is the only still relevant set of instructions for a GPU-based machine.
[Go to the directory](piz-daint)
gpaw/build/examples/davide/gpaw-cpu/build.sh deleted 100644 → 0
View file @ 420d3ef6
### GPAW installation script for D.A.V.I.D.E
# version numbers (modify if needed)
gpaw_version=1.1.0
# installation directory (modify!)
tgt=$CINECA_SCRATCH/lib/gpaw-${gpaw_version}
# setup build environment
module load cuda/9.2.88
module load cudnn/7.1.4--cuda--9.2.88
module load gnu/6.4.0
module load openmpi/3.1.0--gnu--6.4.0
source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
source $CINECA_SCRATCH/lib/python-2018-12-cuda/load.sh 2016-06
source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
export CFLAGS=""
# gpaw
git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
cd gpaw-$gpaw_version
git checkout $gpaw_version
patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
cp ../setup/customize-power8.py .
python setup.py install --customize=customize-power8.py --prefix=$tgt 2>&1 | tee loki-inst
cd ..
sed -e "s|<BASE>|$tgt|g" -e "s|<PYTHONHOME>|$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
# fix permissions
chmod -R g+rwX $tgt
chmod -R o+rX $tgt
gpaw/build/examples/davide/gpaw-cpu/setup/customize-power8.py deleted 100644 → 0
View file @ 420d3ef6
# Setup customisation for gpaw/cuda
import os
# compiler and linker
compiler = 'gcc'
mpicompiler = 'mpicc'
mpilinker = 'mpicc'
extra_compile_args = ['-std=c99', '-mcpu=power8']
# libraries
libraries = ['z']
# openblas
library_dirs += [os.environ['OPENBLAS_ROOT'] + '/lib']
include_dirs += [os.environ['OPENBLAS_ROOT'] + '/include']
libraries += ['openblas']
# scalapack
library_dirs += [os.environ['SCALAPACK_ROOT'] + '/lib']
libraries += ['scalapack']
# libxc
library_dirs += [os.environ['LIBXCDIR'] + '/lib']
include_dirs += [os.environ['LIBXCDIR'] + '/include']
libraries += ['xc']
# GPAW defines
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]
define_macros += [("GPAW_MPI2",1)]
define_macros += [('GPAW_CUDA', '1')]
# ScaLAPACK
scalapack = True
# HDF5
hdf5 = False
gpaw/build/examples/davide/gpaw-cpu/setup/load-gpaw.sh deleted 100644 → 0
View file @ 420d3ef6
#!/bin/bash
module load cuda/9.2.88
module load cudnn/7.1.4--cuda--9.2.88
module load gnu/6.4.0
module load openmpi/3.1.0--gnu--6.4.0
source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
source <PYTHONHOME>/load.sh
source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
export PATH=<BASE>/bin:$PATH
export PYTHONPATH=<BASE>/lib/python2.7/site-packages:$PYTHONPATH
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment