Skip to content
Commits on Source (6)
Expand all Hide whitespace changes
Inline Side-by-side
gpaw/README.md
View file @ b57ca544
......@@ -2,7 +2,7 @@
## Summary version
0.1
0.9
## Purpose of the benchmark
......@@ -12,8 +12,8 @@ augmented wave method. It uses a uniform real-space grid representation of the
electronic wavefunctions that allows for excellent computational scalability
and systematic converge properties.
The GPAW benchmark tests MPI parallelization and the quality of the provided mathematical
libraries, including BLAS, LAPACK, ScaLAPACK, and FFTW-compatible library. There is
The GPAW benchmark tests MPI parallelization and the quality of the provided mathematical
libraries, including BLAS, LAPACK, ScaLAPACK, and FFTW-compatible library. There is
also a CUDA-based implementation for GPU systems.
......@@ -22,13 +22,13 @@ also a CUDA-based implementation for GPU systems.
GPAW is written mostly in Python, but includes also computational kernels
written in C as well as leveraging external libraries such as NumPy, BLAS and
ScaLAPACK. Parallelisation is based on message-passing using MPI with no
support for multithreading.
support for multithreading.
There have been various developments for GPGPUs and MICs in the past
using either CUDA or pyMIC/libxstream. Many of those branches see no
development anymore. The relevant CUDA version for this benchmark is available in a
[separate GitLab for CUDA development, cuda branch](https://gitlab.com/mlouhivu/gpaw/tree/cuda).
This version corresponds to the Aalto version mentioned on the
This version corresponds to the Aalto version mentioned on the
[GPU page of the GPAW Wiki](https://wiki.fysik.dtu.dk/gpaw/devel/projects/gpu.html).
As of early 2020, that version seems to be derived from the 1.5.2 CPU version
(at least, I could find a commit that claims to merge the 1.5.2 code).
......@@ -37,16 +37,23 @@ There is currently no active support for non-CUDA accelerator platforms.
For the UEABS benchmark version 2.2, the following versions of GPAW were tested:
* CPU-based:
* Version 1.5.2 as this one is the last of the 1.5 branch and since the GPU version
is derived from this version.
* Version 20.1.0, the most recent version during the development of the UEABS
2.2 benchmark suite.
* GPU-based: There is no official release or version number. The UEABS 2.2 benchmark
suite was tested using commit TODO of
[the cuda branch of the GitLab for CUDA development](https://gitlab.com/mlouhivu/gpaw/tree/cuda).
* Version 20.1.0, as this one is the version on which the most recent GPU commits
are based.
* Version 20.10.0, as it was the most recent version during the development of
the IEABS 2.2. benchmark suite.
* GPU-based: As there is no official release of the GPU version and as it is
at the moment of the release of the UEABS version 2.2 under heavy development
to also support AMD GPUs, there is no official support for the GPU version
([the cuda branch of the GitLab for CUDA development](https://gitlab.com/mlouhivu/gpaw/tree/cuda))
in UEABS version 2.2.
Versions 1.5.2 and 19.8.1 were also considered but are not compatible with the regular
input files provided here. Hence support for those versions of GPAW was dropped in
this version of the UEABS.
There are three benchmark cases, denotes S, M and L.
### Case S: Carbon nanotube
A ground state calculation for a carbon nanotube in vacuum. By default uses a
......@@ -57,21 +64,25 @@ on 100 cores of a modern x86 cluster.
Input file: [benchmark/1_S_carbon-nanotube/input.py](benchmark/1_S_carbon-nanotube/input.py)
This input file still works with version 1.5.2 and 19.8.1 of GPAW.
### Case M: Copper filament
A ground state calculation for a copper filament in vacuum. By default uses a
3x4x4 FCC lattice with 71 atoms (freely adjustable through the variables `x`,
`y` and `z` in the input file) and ScaLAPACK for
parallellisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
parallelisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
Input file: [benchmark/2_M_copper-filament/input.py](benchmark/2_M_copper-filament/input.py)
The benchmark was tested using 1000 and 1024 cores. For some core configurations, one may
get error messages similar to ``gpaw.grid_descriptor.BadGridError: Grid ... to small
for ... cores``. If one really wants to run the benchmark for those number of cores,
one needs to adapt the values of `x`, `y` and `z` in `input.py`. However, this
changes the benchmark so results cannot be compared easily with benchmark runs for
different values of these variables.
This input file does not work with GPAW 1.5.2 and 19.8.1. It requires GPAW
20.1.0 or 20.10.0. Please try older versions of the UEABS if you want to use
these versions of GPAW.
The benchmark runs best when using full nodes. Expect a
performance drop on other configurations.
### Case L: Silicon cluster
......@@ -82,11 +93,17 @@ and/or 10000 MPI tasks.
Input file: [benchmark/3_L_silicon-cluster/input.py](benchmark/3_L_silicon-cluster/input.py)
This input file does not work with GPAW 1.5.2 and 19.8.1. It requires GPAW
20.1.0 or 20.10.0. Please try older versions of the UEABS if you want to use
these versions of GPAW.
## Mechanics of building the benchmark
Note that GPAW version numbering changed in 2019. Version 1.5.3 is the
last version with the old numbering. In 2019 the development team switched
Installing and running GPAW has changed a lot in the since the previous
versions of the UEABS. GPAW version numbering changed in 2019. Version 1.5.3 is the
last version with the old numbering. In 2019 the development team switched
to a version numbering scheme based on year, month and patchlevel, e.g.,
19.8.1 for the second version released in August 2019.
......@@ -94,22 +111,17 @@ Another change is in the Python packages used to install GPAW. Versions up to
and including 19.8.1 use the `distutils` package while versions 20.1.0 and later
are based on `setuptools`. This does affect the installation process.
GPAW for a while supports two different ways to run in parallel distributed memory mode:
* Using a wrapper executable `gpaw-python` that replaces the Python interpreter (it internally
links to the libpython library) and that provides the MPI functionality.
* Using the standard Python interpreter, including the MPI functionality in the
`_gpaw.so` shared library.
In the `distutils`-based versions, the wrapper script approach is the default behaviour,
while in the `setuptools`-based versions, the approach using the standard Python interpreter
is the preferred one in the manual. Even though the code in the `setuptools`-based
versions still includes the option to use the wrapper script approach, it does not
work in the tested version 20.1.0.
Running GPAW is no longer done via a wrapper executable `gpaw-python` that
replaces the Python interpreter (it internally links to the libpython library)
and that provides the MPI functionality. Since version 20.1.0, the standard Python
interpreter is used and the MPI functionality is included in the `_gpaw.so` shared library.
### Available instructions
The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) only contains the
[installation instructions](https://wiki.fysik.dtu.dk/gpaw/index.html) for the current version.
For the installation instructions with a list of dependencies for older versions,
For the installation instructions with a list of dependencies for older versions,
download the code (see below) and look for the file `doc/install.rst` or go to the
[GPAW GitLab](https://gitlab.com/gpaw), select the tag for the desired version and
view the file `doc/install.rst`.
......@@ -125,47 +137,44 @@ parts and to interface to a number of libraries on which it depends.
Hence GPAW has the following requirements:
* C compiler with MPI support
* BLAS, LAPACK, BLACS and ScaLAPACK. ScaLAPACK is optional for GPAW, but mandatory
for the UEABS benchmarks. It is used by the medium and large cases and optional
* BLAS, LAPACK, BLACS and ScaLAPACK. ScaLAPACK is optional for GPAW, but mandatory
for the UEABS benchmarks. It is used by the medium and large cases and optional
for the small case.
* Python. GPAW 1.5.2 requires
Python 2.7 or 3.4-3.7, GPAW 19.8.1 requires 3.4-3.7, GPAW 20.1.0 Python 3.5-3.8
and GPAW 20.10.0 Python 3.6-3.9.
* Python. GPAW 20.1.0 requires Python 3.5-3.8 and GPAW 20.10.0 Python 3.6-3.9.
* Mandatory Python packages:
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [FFTW](http://www.fftw.org) is highly recommended. As long as the optional libvdwxc
component is not used, the MKL FFTW wrappers can also be used. Recent versions of
GPAW also show good performance using just the NumPy-provided FFT routines provided
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 20.1.0/20.10.0)
GPAW versions before 20.10.0 produce warnings when used with NumPy 1.19.x.
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 20.1.0/20.10.0)
* [FFTW](http://www.fftw.org) is highly recommended. As long as the optional libvdwxc
component is not used, the MKL FFTW wrappers can also be used. Recent versions of
GPAW also show good performance using just the NumPy-provided FFT routines provided
that NumPy has been built with a highly optimized FFT library.
* [LibXC](https://www.tddft.org/programs/libxc/) 2.X or newer for GPAW 1.5.2,
3.X or 4.X for GPAW 19.8.1, 20.1.0 and 20.10.0. LibXC is a library
of exchange-correlation functions for density-functional theory. None of the
versions currently mentions LibXC 5.X as officially supported.
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
* [LibXC](https://www.tddft.org/programs/libxc/) 3.X or 4.X for GPAW 20.1.0 and 20.10.0.
LibXC is a library of exchange-correlation functions for density-functional theory.
None of the versions currently mentions LibXC 5.X as officially supported.
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW
* Check the release notes of GPAW as the releases of ASE and GPAW should match.
E.g., during the development of the UEABS version 2.2 benchamark suite,
version 20.1.0 was the most up-to-date release of GPAW with 3.19.1 the matching ASE version
(though 3.18.0 should also work).
* ASE has some optional dependencies that are not needed for the benchmarking: Matplotlib (2.0.0 or newer),
The benchmarks were tested using ASE 3.19.3 with GPAW 20.1.0 and ASE 3.20.1
with GPAW 20.1.0.
* ASE has some optional dependencies that are not needed for the benchmarking: Matplotlib (2.0.0 or newer),
tkinter (Tk interface, part of the Standard Python Library) and Flask.
* Optional components of GPAW that are not used by the UEABS benchmarks:
* [libvdwxc](https://gitlab.com/libvdwxc/libvdwxc), a portable C library
* [libvdwxc](https://gitlab.com/libvdwxc/libvdwxc), a portable C library
of density functionals with van der Waals interactions for density functional theory.
This library does not work with the MKL FFTW wrappers.
* [ELPA](https://elpa.mpcdf.mpg.de/),
* [ELPA](https://elpa.mpcdf.mpg.de/),
which should improve performance for large systems when GPAW is used in
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
In addition, the GPU version needs:
* NVIDIA CUDA toolkit
* [PyCUDA](https://pypi.org/project/pycuda/)
Installing GPAW also requires a number of standard build tools on the system, including
* [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
* [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
configure script for libxc
* [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
* [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
the configure process of libxc produces very misleading
error messages that do not immediately point to libtool missing.
* [GNU make](https://www.gnu.org/software/make/)
......@@ -173,7 +182,7 @@ Installing GPAW also requires a number of standard build tools on the system, in
### Download of GPAW
GPAW is freely available under the GPL license.
GPAW is freely available under the GPL license.
The source code of the CPU version can be
downloaded from the [GitLab repository](https://gitlab.com/gpaw/gpaw) or as
......@@ -184,7 +193,7 @@ For example, to get version 20.1.0 using git:
git clone -b 20.1.0 https://gitlab.com/gpaw/gpaw.git
```
The CUDA development version is available in
The CUDA development version is available in
[the cuda branch of a separate GitLab](https://gitlab.com/mlouhivu/gpaw/tree/cuda).
To get the current development version using git:
```bash
......@@ -194,17 +203,15 @@ git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
### Install
Crucial for the configuration of GPAW is a proper `customize.py` (GPAW 19.8.1 and
earlier) or `siteconfig.py` (GPAW 20.1.0 and later) file. The defaults used by GPAW
may not offer optimal performance and the automatic detection of the libraries also
Crucial for the configuration of GPAW is a proper `siteconfig.py` file (GPAW 20.1.0 and later,
earlier versions used `customize.py`). The defaults used by GPAW
may not offer optimal performance and the automatic detection of the libraries also
fails on some systems.
The UEABS repository contains additional instructions:
* [general instructions](build/build-cpu.md)
* [GPGPUs](build/build-cuda.md) - To check
Example [build scripts](build/examples/) are also available for some PRACE and non-PRACE
systems.
Example [build scripts](build/examples/) are also available.
## Mechanics of Running the Benchmark
......@@ -216,33 +223,16 @@ right from this repository.
1. [Testcase S: Carbon nanotube input file](benchmark/1_S_carbon-nanotube/input.py)
2. [Testcase M: Copper filament input file](benchmark/2_M_copper-filament/input.py)
3. [Testcase L: Silicon cluster input file](benchmark/3_L_silicon-cluster/input.py)
3. [Testcase L: Silicon cluster input file](benchmark/3_L_silicon-cluster/input.py)
### Running the benchmarks
#### Using the `gpaw-python` wrapper script
This is the default approach for versions up to and including 19.8.1 of GPAW
These instructions are exclusively for GPAW 20.1.0 and later.
These versions of GPAW come with their own wrapper executable, `gpaw-python`,
to start a MPI-based GPAW run.
There are two different ways to start GPAW.
No special command line options or environment variables are needed to run the
benchmarks if your MPI process starter (`mpirun`, Slurm `srun`, ...) communicates
properly with the resource manager. E.g., on Slurm systems, use
```
srun gpaw-python input.py
```
#### Using the regular Python interpreter and parallel GPAW shared library
This is the default method for GPAW 20.1.0 (and likely later).
The wrapper executable `gpaw-python` is no longer available in the default parallel
build of GPAW. There are now two different ways to start GPAW.
One way is through `mpirun`, `srun` or an equivalent process starter and the
One way is through `mpirun`, `srun` or an equivalent process starter and the
`gpaw python` command:
```
srun gpaw python input.py
......@@ -263,13 +253,6 @@ That option however doesn't do the imports in the same way that the `gpaw` scrip
would do.
### Examples
Example [job scripts](scripts/) (`scripts/job-*.sh`) are provided for
different PRACE systems that may offer a helpful starting point.
*TODO: Update the examples as testing on other systems goes on.*
## Verification of Results
### Case S: Carbon nanotube
......
gpaw/build/build-CPU.md
View file @ b57ca544
......@@ -7,25 +7,25 @@ These instructions are in addition to the brief instructions in [README.md](../R
### Libraries and Python interpreter
GPAW needs (for the UEABS benchmarks)
* [Python](https://www.python.org/): GPAW 1.5.2 supports Python 2.7 and 3.4-3.7.
GPAW 19.8.1 needs Python 3.4-3.7 and GPAW 20.1.0 requires Python 3.5-3.8.
* [Python](https://www.python.org/): GPAW 20.1.0 requires Python 3.5-3.8, and
GPAW 20.10.0 and 21.1.0 require Python 3.6-3.9.
* [MPI library](https://www.mpi-forum.org/)
* [LibXC](https://www.tddft.org/programs/libxc/). GPAW 1.5.2 requires LibXC 1.5.2
or later. GPAW 19.8.1 and 20.1.0 need LibXC 3.x or 4.x.
* (Optimized) [BLAS](http://www.netlib.org/blas/) and
[LAPACK](http://www.netlib.org/lapack/) libraries.
There are both commercial and free and open source versions of these libraries.
Using the [reference implementation of BLAS from netlib](http://www.netlib.org/blas/)
* [LibXC](https://www.tddft.org/programs/libxc/). GPAW 20.1.0,
20.10.0 and 21.1.0 all need LibXC 3.x or 4.x.
* (Optimized) [BLAS](http://www.netlib.org/blas/) and
[LAPACK](http://www.netlib.org/lapack/) libraries.
There are both commercial and free and open source versions of these libraries.
Using the [reference implementation of BLAS from netlib](http://www.netlib.org/blas/)
will give very poor performance. Most optimized LAPACK libraries actually only
optimize a few critical routines while the remaining routines are compiled from
optimize a few critical routines while the remaining routines are compiled from
the reference version. Most processor vendors for HPC machines and system vendors
offer optmized versions of these libraries.
* [ScaLAPACK](http://www.netlib.org/scalapack/) and the underlying communication
layer [BLACS](http://www.netlib.org/blacs/).
* [FFTW](http://www.fftw.org/) or compatible FFT library.
For the UEABS benchmarks, the double precision, non-MPI version is sufficient.
* [FFTW](http://www.fftw.org/) or compatible FFT library.
For the UEABS benchmarks, the double precision, non-MPI version is sufficient.
GPAW also works with the
[Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html)
[Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html)
FFT routines when using the FFTW wrappers provided with that product.
For the GPU version, the following packages are needed in addition to the packages
......@@ -34,48 +34,48 @@ above:
* [PyCUDA](https://pypi.org/project/pycuda/)
Optional components of GPAW that are not used by the UEABS benchmarks:
* [libvdwxc](https://gitlab.com/libvdwxc/libvdwxc), a portable C library
* [libvdwxc](https://gitlab.com/libvdwxc/libvdwxc), a portable C library
of density functionals with van der Waals interactions for density functional theory.
This library does not work with the MKL FFTW wrappers as it needs the MPI version
of the FFTW libraries too.
* [ELPA](https://elpa.mpcdf.mpg.de/),
* [ELPA](https://elpa.mpcdf.mpg.de/),
which should improve performance for large systems when GPAW is used in
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
### Python packages
GPAW needs
* [wheel](https://pypi.org/project/wheel/) is needed in most (if not all) ways of
GPAW needs
* [wheel](https://pypi.org/project/wheel/) is needed in most (if not all) ways of
installing the packages from source.
* [NumPy](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* Installing NumPy from source will also require
* [NumPy](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 20.1.0/20.10.0/21.1.0)
* Installing NumPy from source will also require
[Cython](https://pypi.org/project/Cython/)
* GPAW 1.5.2 is not fully compatible with NumPy 1.19.x. Warnings about the use
* GPAW 20.1.0 is not fully compatible with NumPy 1.19.x or later. Warnings about the use
of deprecated constructs will be shown.
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW. Required versions are 3.17.0 or later for
GPAW 1.5.2 and 3.18.0 or later for GPAW 19.8.1 or 20.1.0.
ASE has a couple of dependendencies
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 20.1.0/20.10.0/21.1.0)
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW. The required versions is 3.18.0 or later for
GPAW 20.1.0, 20.10.0 and 21.1.0.
ASE has a couple of dependendencies
that are not needed for running the UEABS benchmarks. However, several Python
package install methods will trigger the installation of those packages, and
package install methods will trigger the installation of those packages, and
with them may require a chain of system libraries.
* ASE does need NumPy and SciPy, but these are needed anyway for GPAW.
* [matplotlib](https://pypi.org/project/matplotlib/), at least version 2.0.0.
This package is optional and not really needed to run the benchmarks.
Matplotlib pulls in a lot of other dependencies. When installing ASE with pip,
it will try to pull in matplotlib and its dependencies
* [pillow](https://pypi.org/project/Pillow/) needs several exgternal
* [pillow](https://pypi.org/project/Pillow/) needs several exgternal
libraries. During the development of the benchmarks, we needed at least
zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even
though the pillow documentation claimed that libjpeg was optional,
zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even
though the pillow documentation claimed that libjpeg was optional,
it refused to install without.
* [kiwisolver](https://pypi.org/project/kiwisolver/): Contains C++-code
* [pyparsing](https://pypi.org/project/pyparsing/)
* [Cycler](https://pypi.org/project/Cycler/), which requires
* [six](https://pypi.org/project/six/)
* [python-dateutil](https://pypi.org/project/python-dateutil/), which also
* [python-dateutil](https://pypi.org/project/python-dateutil/), which also
requires
* [six](https://pypi.org/project/six/)
* [Flask](https://pypi.org/project/Flask/) is an optional dependency of ASE
......@@ -83,7 +83,7 @@ GPAW needs
the development of this version of the UEABS. It has a number of dependencies
too:
* [Jinja2](https://pypi.org/project/Jinja2/)
* [MarkupSafe](https://pypi.org/project/MarkupSafe/), contains some C
* [MarkupSafe](https://pypi.org/project/MarkupSafe/), contains some C
code
* [itsdangerous](https://pypi.org/project/itsdangerous/)
* [Werkzeug](https://pypi.org/project/Werkzeug/)
......@@ -116,24 +116,24 @@ GPAW needs
* ASE
* GPAW
The table below give the combinations of major packages Python, NumPy, SciPy, ASE and
The table below give the combinations of major packages Python, NumPy, SciPy, ASE and
GPAW that were tested:
| Python | NumPy | SciPy | ASE | GPAW |
|:-------|:-------|:------|:-------|:--------|
| 3.7.9 | 1.18.5 | 1.4.1 | 3.17.0 | 1.5.2 |
| 3.7.9 | 1.18.5 | 1.4.1 | 3.18.2 | 19.8.1 |
| 3.8.6 | 1.18.5 | 1.4.1 | 3.19.3 | 20.1.0 |
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.1 | 1.19.5 | 1.5.4 |
| 21.1.0 | 3.21.1 | 3.9.1 | 1.19.5 | 1.5.4 |
## Installing all prerequisites
We do not include the optimized mathematical libraries in the instructions (BLAS, LAPACK,
FFT library, ...) as these libraries should be standard on any optimized HPC system.
Also, the instructions below will need to be adapted to the specific
FFT library, ...) as these libraries should be standard on any optimized HPC system.
Also, the instructions below will need to be adapted to the specific
libraries that are being used.
Other prerequisites:
Other prerequisites:
* libxc
* Python interpreter
* Python package NumPy
......@@ -143,11 +143,11 @@ Other prerequisites:
### Installing libxc
* Installing libxc requires GNU automake and GNU buildtool besides GNU make and a
C compiler. The build process is the usual GNU configure - make - make install
* Installing libxc requires GNU automake and GNU buildtool besides GNU make and a
C compiler. The build process is the usual GNU configure - make - make install
cycle, but the `configure` script still needs to be generated with autoreconf.
* Download libxc:
* The latest version of libxc can be downloaded from
* The latest version of libxc can be downloaded from
[the libxc download page](https://www.tddft.org/programs/libxc/download/).
However, that version may not be officially supported by GPAW.
* It is also possible to download all recent versions of libxc from
......@@ -157,14 +157,14 @@ Other prerequisites:
* Then use the download button and select the desired file type.
* Dowload URLs look like `https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2`.
* Untar the file in the build directory.
### Installing Python from scratch
The easiest way to get Python on your system is to download an existing distribution
(one will likely already be installed on your system). Python itself does have a lot
(one will likely already be installed on your system). Python itself does have a lot
of dependencies though, definitely in its Standard Python Library. Many of the
standard packages are never needed when executing the benchmark cases. Isolating them
to compile a Python with minimal dependencies is beyond the scope though. We did
......@@ -179,8 +179,8 @@ of NumPy, SciPy and GPAW itself proves much more important.
### Installing NumPy
* As NumPy relies on optimized libraries for its performance, one should carefully
select which NumPy package to download, or install NumPy from sources. How crucial
this is, depends on the version of GPAW and the options selected when building
select which NumPy package to download, or install NumPy from sources. How crucial
this is, depends on the version of GPAW and the options selected when building
GPAW.
* Given that GPAW also uses optimized libraries, it is generally advised to install
NumPy from sources instead to ensure that the same libraries are used as will be
......@@ -193,11 +193,11 @@ of NumPy, SciPy and GPAW itself proves much more important.
### Installing SciPy
* Just as NumPy, SciPy relies on optimized libraries for its performance. It should
be installed after NumPy as it does get the information about which libraries to
be installed after NumPy as it does get the information about which libraries to
use from NumPy. Hence, when installing pre-built binaries, make sure they match
the NumPy binaries used.
* Just as is the case for NumPy, it may be better to install SciPy from sources.
[Instructions for installing SciPy from source can be found on the SciPy GitHub
[Instructions for installing SciPy from source can be found on the SciPy GitHub
site](https://github.com/scipy/scipy/blob/master/INSTALL.rst.txt).
......@@ -205,7 +205,7 @@ of NumPy, SciPy and GPAW itself proves much more important.
* Just as for any user-installed Python package, make sure you have created a
directory to install Python packages to and have added it to the front of PYTHONPATH.
* ase is [available on PyPi](https://pypi.org/project/ase/). It is also possible
* ase is [available on PyPi](https://pypi.org/project/ase/). It is also possible
to [see a list of previous releases](https://pypi.org/project/ase/#history).
* The easiest way to install ase is using `pip` which will automatically download.
the requested version.
......@@ -213,52 +213,18 @@ of NumPy, SciPy and GPAW itself proves much more important.
## Configuring and installing GPAW
### GPAW 1.5.2
* GPAW 1.5.2 uses `distutils`. Customization of the installation process is possible
through the `customize.py` file.
* The FFT library: According to the documentation, the following strategy is used
* The compile process searches (in this order) for ``libmkl_rt.so``,
``libmkl_intel_lp64.so`` and ``libfftw3.so`. First one found will be
loaded.
* If none is found, the built-in FFT from NumPy will be used. This does not need
to be a problem if NumPy provides a properly optimized FFT library.
* The choice can also be overwritten using the GPAW_FFTWSO environment variable.
* With certain compilers, the GPAW test suite produced crashes in `xc/xc.py`. The
patch for GPAW 1.5.2 included in the [pathces](patches) subdirectory solved these
problems on the systems tested.
### GPAW 19.8.1
* GPAW 19.8.1 uses `distutils`. Customization of the installation process is possible
through a `customize.py` file.
* The selection process of the FFT library has changed from version 1.5.2. It is
now possible to specify the FFT library in `customize.py` or to simply select to
* GPAW 20.1.0 and later use `setuptools`. Customization of the installation process is possible
through the `siteconfig.py` file.
* It is possible to specify the FFT library in `siteconfrig.py or to simply select to
use the NumPy FFT routines.
### GPAW 20.1.0 and 20.10.0
* GPAW 20.1.0 uses `setuptools`. Customization of the installation process is possible
through the `siteconfig.py` file.
* The selection process of the FFT library is the same as in version 19.8.1, except
that the settings are now in `siteconfrig.py` rather than `customize.py`.
### All versions
* GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files
* GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files
can be found on the [GPAW website, Atomic PAW Setups page](https://wiki.fysik.dtu.dk/gpaw/setups/setups.html).
For the testing we used []`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
for all versions of GPAW. The easiest way to install these files is to simpy untar
the file and set the environment variable GPAW_SETUP_PATH to point to that directory.
In the examples provided we use the `share/gpaw-setups` subdirectory of the install
directory for this purpose.
* Up to and including version 20.1.0, GPAW does comes with a test suite which can be
used after installation.
* GPAW 20.1.0 comes with a test suite which can be used after installation.
* Running the sequential tests:
gpaw test
......@@ -271,7 +237,7 @@ of NumPy, SciPy and GPAW itself proves much more important.
gpaw test -j 4
We did experience that crashed that cause segmentation faults get unnoticed
We did experience that crashed that cause segmentation faults get unnoticed
in this setup. They are not mentioned as failed.
* Running the parallel benchmarks on a SLURM cluster will depend on the version of GPAW.
......@@ -288,7 +254,7 @@ of NumPy, SciPy and GPAW itself proves much more important.
* Depending on the Python installation, some tests may fail with error messages that point
to a package in the Standard Python Library that is not present. Some of these errors have no
influence on the benchmarks as that part of the code is not triggered by the benchmark.
* The full test suite is missing in GPAW 20.10.0. There is a brief sequential test
* The full test suite is missing in GPAW 20.10.0 and later. There is a brief sequential test
that can be run with
gpaw test
......@@ -297,21 +263,21 @@ of NumPy, SciPy and GPAW itself proves much more important.
gpaw -P 4 test
* Multiple versions of GPAW likely contain a bug in `c/bmgs/fd.c` (around line 44
in GPAW 1.5.2). The code enforces vectorization on OpenMP 4 compilers by using
`#pragma omp simd`. However, it turns out that the data is not always correctly
* Multiple versions of GPAW likely contain a bug in `c/bmgs/fd.c` (around line 44
in GPAW 20.1.0). The code enforces vectorization on OpenMP 4 compilers by using
`#pragma omp simd`. However, it turns out that the data is not always correctly
aligned, so if the reaction of the compiler to `#pragma omp simd` is to fully vectorize
and use load/store instructions for aligned data, crashes may occur. It did happen
during the benchmark development when compiling with the Intel C compiler. The
and use load/store instructions for aligned data, crashes may occur. It did happen
during the benchmark development when compiling with the Intel C compiler. The
solution for that compiler is to add `-qno-openmp-simd` to the compiler flags.
## Problems observed during testing
* On AMD Epyc systems, there seems to be a bug in the Intel MKL FFT libraries/FFTW
wrappers in the 2020 compilers. Downgrading to the MKL libraries of the 2018
* On AMD Epyc systems, there seems to be a bug in the Intel MKL FFT libraries/FFTW
wrappers in the 2020 compilers. Downgrading to the MKL libraries of the 2018
compilers or using the FFTW libraries solves the problem.
This has been observed not only in GPAW, but also in some other DFT packages.
* The GPAW test code in versions 1.5.2 till 20.1.0 detects that matplotlib is not installed
* The GPAW test code in version 20.1.0 detects if matplotlib is not installed
and will skip this test. We did however observe a failed test when Python could not find
the SQLite package as the Python standard library sqlite3 package is used.
gpaw/build/examples/CalcUA-leibniz-broadwell/README.md 0 → 100644
View file @ b57ca544
# Example build scripts for CalcUA-leibniz-broadwell
Cluster characteristics:
* CPU architecture: dual-socket 14-core Intel Xeon E5-2680v4 "broadwell" CPUs
* Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
* Package manager: Most software on the cluster is installed through [
EasyBuild](https://easybuilders.github.io/).
Unless otherwise mentioned below, the following combinations of versions of GPAW, ASE, Python, NumPy
and SciPy were used:
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.1 | 1.19.5 | 1.5.4 |
| 21.1.0 | 3.21.1 | 3.9.1 | 1.19.5 | 1.5.4 |
The following example build scripts do a complete build, including a build of Python
and a minimal set of dependencies that is needed to build a large enough part of the
Python standard library to build GPAW and run the benchmarks: zlib,
ncurses, readline, libffi, SQLite (used by one of the GPAW tests but likely
not needed for the benchmarks), FFTW, libxc, Python and the Python packages
NumPy, SciPy, ase and GPAW. The included libraries are not sufficient for a
full Python installation with all standard library packages or a full ASE/GPAW
installation with all optional packages but should be enough to run the benchmarks.
All software is build with the Intel 2020 compilers.
* `build_20.1.0_Python38_FFTW_icc`
* `build_20.10.0_Python39_FFTW_icc`
* `build_21.1.0_Python39_FFTW_icc`
The following build example build scripts install GPAW on top of the Intel Python
distribution. They rely as much as possible on modules already in the Intel Python
distribution or software that should be installed in the OS to run Intel Python.
The configuration accesses the FFT library available through
NumPy rather then linking directly to a FFT library.
The example script does use a module that provides up-to-date build
tools (buildtools/2020a) but on recent OSes it can be omitted.
Rather than loading the Intel and IntelPython3 modules one could as well
run the scripts provided by Intel to initialize the environment variables
for the compilers and Python distribution, and provided the necessary build
tools are also available in the system OS, this build could be done without
relying on any EasyBuild or Spack-generated modules.
* `build_20.1.0_IntelPython3_icc.sh`
* `build_20.10.0_IntelPython3_icc.sh`
* `build_21.1.0_IntelPython3_icc.sh`
gpaw/build/examples/CalcUA-leibniz-broadwell/build_20.10.0_IntelPython3_icc.sh 0 → 100755
View file @ b57ca544
#!/bin/bash
#
# Installation script for GPAW 20.10.0:
# * Using the existing IntelPython3 module on the system which has an optimized
# NumPy and SciPy included.
# * Using the matching version of ase, 3.20.1
# * Compiling with the Intel compilers
#
# The FFT library is discovered at runtime. With the settings used in this script
# this should be MKL FFT, but it is possible to change this at runtime to either
# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
# (link below).
#
# The original installation instructions for GPAW can be found at
# https://gitlab.com/gpaw/gpaw/-/blob/20.10.0/doc/install.rst
#
packageID='20.10.0-IntelPython3-icc'
echo -e "\n### Building GPAW-UEABS/$packageID from $0\n\n"
install_root=$VSC_SCRATCH/UEABS
systemID=CalcUA-leibniz-broadwell
UEABS_version='2.2'
#
# NOTE: For easy maintanance we load many settings from the UEABS module.
# Since we want to release the module again and only depend on system modules for
# this build, we copy the settings to other variables and unload the module
# again. It is of course possible to those variables directly by hand without
# first creating a module file.
#
module purge
module use $install_root/$systemID/Modules
module load UEABS/$UEABS_version
# Directory in which downloaded files will be stored.
download_dir=$UEABS_DOWNLOADS
# Directory in which the software package will be installed.
install_dir=$UEABS_PACKAGES/GPAW-UEABS/$packageID
# Subdirectory containing the module file.
modules_dir=$UEABS_MODULES/GPAW-UEABS
# Temporary directory for the build.
build_dir="/dev/shm/$USER/GPAW-UEABS/$packageID"
# Subdirectory in the UEABS repo containing the patch file that is needed.
patch_dir=$UEABS_REPOSITORY/gpaw/build/patches
#
# Software versions
#
IntelPython3_version='2020.02'
libxc_version='4.3.4'
ase_version='3.20.1'
GPAW_version='20.10.0'
GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
# Compiler settings
compiler_module='intel/2020a'
opt_level='-O2'
proc_opt_flags='-march=core-avx2 -mtune=core-avx2'
fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
parallel=32
py_maj_min='3.7'
################################################################################
#
# Prepare the system
#
# We do this through modules, but it could be done by running scripts from the
# Intel compiler suite as well.
#
#
# Load modules:
# - Intel compiler module
# - Intel Python
# - buildtools is a module with a number of build tools that are more up-to-date
# then those on the system. It is mostly used for consistent builds across
# systems.
#
module purge
MODULEPATH=/apps/antwerpen/modules/centos7/calcua-admin-broadwell/
module load calcua/admin
module load $compiler_module
module load buildtools/2020a
module load IntelPython3/$IntelPython3_version
#
# Create the directories and make sure they are clean if that matters
#
/usr/bin/mkdir -p $download_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/rm -rf $install_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/mkdir -p $modules_dir
/usr/bin/mkdir -p $build_dir
/usr/bin/rm -rf $build_dir
/usr/bin/mkdir -p $build_dir
################################################################################
#
# Download components
#
echo -e "\nDownloading files...\n"
cd $download_dir
# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
libxc_file="libxc-$libxc_version.tar.bz2"
libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
# Downloading ase so that we can gather all sources for reproducibility
ase_file="ase-$ase_version.tar.gz"
ase_url="https://pypi.python.org/packages/source/a/ase"
[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# GPAW needs customization, so we need to download and unpack the sources.
# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
GPAW_file="gpaw-$GPAW_version.tar.gz"
GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
# Download GPAW-setup, a number of setup files for GPAW.
# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
################################################################################
#
# Install libxc
#
echo -e "\nInstalling libxc...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$libxc_file
cd libxc-$libxc_version
# Configure
autoreconf -i
export CC=icc
export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
./configure --prefix="$install_dir" \
--disable-static --enable-shared --disable-fortran
# Build
make -j $parallel
# Install
make install
# Add bin, lib and include to the PATH variables
PATH=$install_dir/bin:$PATH
LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
CPATH=$install_dir/include:$CPATH
# Clean-up
unset CC
unset CFLAGS
################################################################################
#
# Prepare for installing Python packages
#
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages:$PYTHONPATH"
################################################################################
#
# Optional: Install pytest and its dependencies to test NumPy and SciPy with
# import numpy
# numpy.test()
# import scipy
# scipy.text()
# We don't care about version numbers here as it is not important for the
# reproducibility of the benchmarks.
#
echo -e "\nInstalling pytest...\n"
cd $build_dir
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir pytest
################################################################################
#
# Install ase
#
echo -e "\nInstalling ase...\n"
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir --no-deps $download_dir/$ase_file
# Brief test
cd $build_dir
python$py_maj_min -c "import ase"
################################################################################
#
# Install GPAW-setups
#
echo -e "\nInstalling gpaw-setups...\n"
mkdir -p $install_dir/share/gpaw-setups
cd $install_dir/share/gpaw-setups
tar -xf $download_dir/$GPAWsetups_file --strip-components=1
################################################################################
#
# Install GPAW
#
echo -e "\nInstalling GPAW...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$GPAW_file
cd gpaw-$GPAW_version
# Make the customize.py script
cat >siteconfig.py <<EOF
print( 'GPAW EasyBuild INFO: Starting execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the start of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
# LibXC
include_dirs.append('$install_dir/include')
#libraries.append('xc')
# libvdwxc
libvdwxc = False
# ELPA
elpa = False
# Use NumPy FFTW
fftw = False
# ScaLAPACK
scalapack = True
libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
# MKL BLAS
libraries += ['mkl_sequential','mkl_core', 'mkl_rt', ]
# Add other EasyBuild library directoryes.
library_dirs = os.environ['LIBRARY_PATH'].split(':')
# Set the compilers
compiler = os.environ['CC']
mpicompiler = os.environ['MPICC']
mpilinker = os.environ['MPICC']
print( 'GPAW EasyBuild INFO: Variables at the end of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
EOF
# Now install gpaw
export CC=icc
export MPICC=mpiicc
export CFLAGS="-std=c99 $opt_level $proc_opt_flags $fp_opt_flags -qno-openmp-simd"
python$py_maj_min setup.py build -j $parallel
# Install GPAW
pip install --prefix=$install_dir --no-deps --ignore-installed --no-build-isolation .
# Brief test
cd $build_dir
python$py_maj_min -c "import gpaw"
# Clean-up
unset CC
unset MPICC
unset CFLAGS
################################################################################
#
# Finish the install
#
echo -e "\nCleaning up and making the LUA-module GPAW-UEABS/$packageID...\n"
# Go to a different directory before cleaning up the build directory
cd $modules_dir
/bin/rm -rf $build_dir
# Create a module file
python_version=$(python -V | cut -d ' ' -f 2)
numpy_version=$(python3 -c "import numpy ; print(numpy.__version__)")
scipy_version=$(python3 -c "import scipy ; print(scipy.__version__)")
cat >$packageID.lua <<EOF
help([==[
Description
===========
GPAW $GPAW_version for the UEABS benchmark.
Configuration:
* Parallel GPAW $GPAW_version with ase $ase_version
* On top of the system IntelPython3 $IntelPython3_version module, which provides
Python $python_version, NumPy $numpy_version and SciPy $scipy_version.
* NumPy is used for FFT, so the performance of the code
will depend a lot on a proper configuration of NumPy for FFT.
* libxc and GPAW compiled with the Intel compilers
Detailed configuration:
* Compiler module: $compiler_module (a module file written
at UAntwerp, not generated via EasyBuild)
* Modules from the system
* Intel Python3 $IntelPython3_version (using a module file written
at UAntwerp)
* libxc $libxc_version
* Python packages: ase-$ase_version, gpaw-$GPAW_version
* GPAW setups $GPAWsetups_version
More information
================
- Homepage: http://wiki.fysik.dtu.dk/gpaw
- Documentation:
- GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
- Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
- ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version, Intel Python3 $IntelPython3_version with numpy $numpy_version and scipy $scipy_version: UEABS benchmark configuration.]==])
family("GPAW")
prepend_path('MODULEPATH','/apps/antwerpen/modules/centos8/calcua-admin-rome')
if not ( isloaded("calcua/admin") ) then
load("calcua/admin")
end
if not ( isloaded("$compiler_module") ) then
load("$compiler_module")
end
if not ( isloaded("IntelPython3/$IntelPython3_version") ) then
load("IntelPython3/$IntelPython3_version")
end
prepend_path("PATH", "$install_dir/bin")
prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
prepend_path("LIBRARY_PATH", "$install_dir/lib")
prepend_path("PYTHONPATH", "$install_dir/lib/python$py_maj_min/site-packages")
setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
EOF
gpaw/build/examples/CalcUA-leibniz-broadwell/build_21.1.0_IntelPython3_icc.sh 0 → 100755
View file @ b57ca544
#!/bin/bash
#
# Installation script for GPAW 21.1.0:
# * Using the existing IntelPython3 module on the system which has an optimized
# NumPy and SciPy included.
# * Using the matching version of ase, 3.21.1
# * Compiling with the Intel compilers
#
# The FFT library is discovered at runtime. With the settings used in this script
# this should be MKL FFT, but it is possible to change this at runtime to either
# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
# (link below).
#
# The original installation instructions for GPAW can be found at
# https://gitlab.com/gpaw/gpaw/-/blob/21.1.0/doc/install.rst
#
packageID='21.1.0-IntelPython3-icc'
echo -e "\n### Building GPAW-UEABS/$packageID from $0\n\n"
install_root=$VSC_SCRATCH/UEABS
systemID=CalcUA-leibniz-broadwell
UEABS_version='2.2'
#
# NOTE: For easy maintanance we load many settings from the UEABS module.
# Since we want to release the module again and only depend on system modules for
# this build, we copy the settings to other variables and unload the module
# again. It is of course possible to those variables directly by hand without
# first creating a module file.
#
module purge
module use $install_root/$systemID/Modules
module load UEABS/$UEABS_version
# Directory in which downloaded files will be stored.
download_dir=$UEABS_DOWNLOADS
# Directory in which the software package will be installed.
install_dir=$UEABS_PACKAGES/GPAW-UEABS/$packageID
# Subdirectory containing the module file.
modules_dir=$UEABS_MODULES/GPAW-UEABS
# Temporary directory for the build.
build_dir="/dev/shm/$USER/GPAW-UEABS/$packageID"
# Subdirectory in the UEABS repo containing the patch file that is needed.
patch_dir=$UEABS_REPOSITORY/gpaw/build/patches
#
# Software versions
#
IntelPython3_version='2020.02'
libxc_version='4.3.4'
ase_version='3.21.1'
GPAW_version='21.1.0'
GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
# Compiler settings
compiler_module='intel/2020a'
opt_level='-O2'
proc_opt_flags='-march=core-avx2 -mtune=core-avx2'
fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
parallel=32
py_maj_min='3.7'
################################################################################
#
# Prepare the system
#
# We do this through modules, but it could be done by running scripts from the
# Intel compiler suite as well.
#
#
# Load modules:
# - Intel compiler module
# - Intel Python
# - buildtools is a module with a number of build tools that are more up-to-date
# then those on the system. It is mostly used for consistent builds across
# systems.
#
module purge
MODULEPATH=/apps/antwerpen/modules/centos7/calcua-admin-broadwell/
module load calcua/admin
module load $compiler_module
module load buildtools/2020a
module load IntelPython3/$IntelPython3_version
#
# Create the directories and make sure they are clean if that matters
#
/usr/bin/mkdir -p $download_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/rm -rf $install_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/mkdir -p $modules_dir
/usr/bin/mkdir -p $build_dir
/usr/bin/rm -rf $build_dir
/usr/bin/mkdir -p $build_dir
################################################################################
#
# Download components
#
echo -e "\nDownloading files...\n"
cd $download_dir
# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
libxc_file="libxc-$libxc_version.tar.bz2"
libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
# Downloading ase so that we can gather all sources for reproducibility
ase_file="ase-$ase_version.tar.gz"
ase_url="https://pypi.python.org/packages/source/a/ase"
[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# GPAW needs customization, so we need to download and unpack the sources.
# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
GPAW_file="gpaw-$GPAW_version.tar.gz"
GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
# Download GPAW-setup, a number of setup files for GPAW.
# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
################################################################################
#
# Install libxc
#
echo -e "\nInstalling libxc...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$libxc_file
cd libxc-$libxc_version
# Configure
autoreconf -i
export CC=icc
export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
./configure --prefix="$install_dir" \
--disable-static --enable-shared --disable-fortran
# Build
make -j $parallel
# Install
make install
# Add bin, lib and include to the PATH variables
PATH=$install_dir/bin:$PATH
LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
CPATH=$install_dir/include:$CPATH
# Clean-up
unset CC
unset CFLAGS
################################################################################
#
# Prepare for installing Python packages
#
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages:$PYTHONPATH"
################################################################################
#
# Optional: Install pytest and its dependencies to test NumPy and SciPy with
# import numpy
# numpy.test()
# import scipy
# scipy.text()
# We don't care about version numbers here as it is not important for the
# reproducibility of the benchmarks.
#
echo -e "\nInstalling pytest...\n"
cd $build_dir
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir pytest
################################################################################
#
# Install ase
#
echo -e "\nInstalling ase...\n"
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir --no-deps $download_dir/$ase_file
# Brief test
cd $build_dir
python$py_maj_min -c "import ase"
################################################################################
#
# Install GPAW-setups
#
echo -e "\nInstalling gpaw-setups...\n"
mkdir -p $install_dir/share/gpaw-setups
cd $install_dir/share/gpaw-setups
tar -xf $download_dir/$GPAWsetups_file --strip-components=1
################################################################################
#
# Install GPAW
#
echo -e "\nInstalling GPAW...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$GPAW_file
cd gpaw-$GPAW_version
# Make the customize.py script
cat >siteconfig.py <<EOF
print( 'GPAW EasyBuild INFO: Starting execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the start of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
# LibXC
include_dirs.append('$install_dir/include')
#libraries.append('xc')
# libvdwxc
libvdwxc = False
# ELPA
elpa = False
# Use NumPy FFTW
fftw = False
# ScaLAPACK
scalapack = True
libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
# MKL BLAS
libraries += ['mkl_sequential','mkl_core', 'mkl_rt', ]
# Add other EasyBuild library directoryes.
library_dirs = os.environ['LIBRARY_PATH'].split(':')
# Set the compilers
compiler = os.environ['CC']
mpicompiler = os.environ['MPICC']
mpilinker = os.environ['MPICC']
print( 'GPAW EasyBuild INFO: Variables at the end of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
EOF
# Now install gpaw
export CC=icc
export MPICC=mpiicc
export CFLAGS="-std=c99 $opt_level $proc_opt_flags $fp_opt_flags -qno-openmp-simd"
python$py_maj_min setup.py build -j $parallel
# Install GPAW
pip install --prefix=$install_dir --no-deps --ignore-installed --no-build-isolation .
# Brief test
cd $build_dir
python$py_maj_min -c "import gpaw"
# Clean-up
unset CC
unset MPICC
unset CFLAGS
################################################################################
#
# Finish the install
#
echo -e "\nCleaning up and making the LUA-module GPAW-UEABS/$packageID...\n"
# Go to a different directory before cleaning up the build directory
cd $modules_dir
/bin/rm -rf $build_dir
# Create a module file
python_version=$(python -V | cut -d ' ' -f 2)
numpy_version=$(python3 -c "import numpy ; print(numpy.__version__)")
scipy_version=$(python3 -c "import scipy ; print(scipy.__version__)")
cat >$packageID.lua <<EOF
help([==[
Description
===========
GPAW $GPAW_version for the UEABS benchmark.
Configuration:
* Parallel GPAW $GPAW_version with ase $ase_version
* On top of the system IntelPython3 $IntelPython3_version module, which provides
Python $python_version, NumPy $numpy_version and SciPy $scipy_version.
* NumPy is used for FFT, so the performance of the code
will depend a lot on a proper configuration of NumPy for FFT.
* libxc and GPAW compiled with the Intel compilers
Detailed configuration:
* Compiler module: $compiler_module (a module file written
at UAntwerp, not generated via EasyBuild)
* Modules from the system
* Intel Python3 $IntelPython3_version (using a module file written
at UAntwerp)
* libxc $libxc_version
* Python packages: ase-$ase_version, gpaw-$GPAW_version
* GPAW setups $GPAWsetups_version
More information
================
- Homepage: http://wiki.fysik.dtu.dk/gpaw
- Documentation:
- GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
- Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
- ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version, Intel Python3 $IntelPython3_version with numpy $numpy_version and scipy $scipy_version: UEABS benchmark configuration.]==])
family("GPAW")
prepend_path('MODULEPATH','/apps/antwerpen/modules/centos8/calcua-admin-rome')
if not ( isloaded("calcua/admin") ) then
load("calcua/admin")
end
if not ( isloaded("$compiler_module") ) then
load("$compiler_module")
end
if not ( isloaded("IntelPython3/$IntelPython3_version") ) then
load("IntelPython3/$IntelPython3_version")
end
prepend_path("PATH", "$install_dir/bin")
prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
prepend_path("LIBRARY_PATH", "$install_dir/lib")
prepend_path("PYTHONPATH", "$install_dir/lib/python$py_maj_min/site-packages")
setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
EOF
gpaw/build/examples/CalcUA-vaughan-rome/README.md 0 → 100644
View file @ b57ca544
# Example build scripts for CalcUA-vaughan-rome
Cluster characteristics:
* CPU architecture: dual-socket 32-core AMD EPYC 7452 "Rome" CPUs
* Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
* Package manager: Most software on the cluster is installed through [
EasyBuild](https://easybuilders.github.io/).
Unless otherwise mentioned below, the following combinations of versions of GPAW, ASE, Python, NumPy
and SciPy were used:
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.1 | 1.19.5 | 1.5.4 |
| 21.1.0 | 3.21.1 | 3.9.1 | 1.19.5 | 1.5.4 |
The following example build scripts do a complete build, including a build of Python
and a minimal set of dependencies that is needed to build a large enough part of the
Python standard library to build GPAW and run the benchmarks: zlib,
ncurses, readline, libffi, SQLite (used by one of the GPAW tests but likely
not needed for the benchmarks), FFTW, libxc, Python and the Python packages
NumPy, SciPy, ase and GPAW. The included libraries are not sufficient for a
full Python installation with all standard library packages or a full ASE/GPAW
installation with all optional packages but should be enough to run the benchmarks.
All software is build with the Intel 2020 compilers.
* `build_20.1.0_Python38_FFTW_icc`
* `build_20.10.0_Python39_FFTW_icc`
* `build_21.1.0_Python39_FFTW_icc`
The following build example build scripts install GPAW on top of the Intel Python
distribution. They rely as much as possible on modules already in the Intel Python
distribution or software that should be installed in the OS to run Intel Python.
The configuration accesses the FFT library available through
NumPy rather then linking directly to a FFT library.
The example script does use a module that provides up-to-date build
tools (buildtools/2020a) but on recent OSes it can be omitted.
Rather than loading the Intel and IntelPython3 modules one could as well
run the scripts provided by Intel to initialize the environment variables
for the compilers and Python distribution, and provided the necessary build
tools are also available in the system OS, this build could be done without
relying on any EasyBuild or Spack-generated modules.
* `build_20.1.0_IntelPython3_icc.sh`
* `build_20.10.0_IntelPython3_icc.sh`
* `build_21.1.0_IntelPython3_icc.sh`
The following example build scripts depend on other modules (Python and FFTW) that
were installed on the cluster through EasyBuild beyond the compiler modules and a
module providing some up-to-date basic build tools:
* `build_20.1.0_Python39icc_icc.sh`
* `build_20.10.0_Python39icc_icc.sh`
* `build_21.1.0_Python39icc_icc.sh`
gpaw/build/examples/README.md 0 → 100644
View file @ b57ca544
# Example build scripts
## CalcUA-vaughan-rome
Vaughan is a cluster installed at the University of Antwerpen
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256GB of RAM memory
per node. Nodes are interconnected through an InfiniBand HDR100 network. The system
does not contain accelerators. The cluster uses CentOS 8 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](CalcUA-vaughan-rome)
## CalcUA-leibniz/broadwell
Leibniz is a cluster intalled at the University of Antwerpen
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 153 regular CPU nodes with dual Intel Xeon E5-2680v4
"broadwell" CPUs with 128 or 256 GB of RAM memory per node.
Nodes are interconnected through an Infiniband EDR network.
The system has two nodes with dual NVIDIA P100 GPUs and a node
with dual NEC Aurora TSUBASA first generation vector boards.
During the testing period the cluster used CentOS 7 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
## davide @ Cineca
Build instructions for GPU and CPU on the DAVIDE OpenPower GPU cluster @ Cineca.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
## juwels @ Jülich Supercomputing Centre
Build instructions for the CPU nodes of JUWELS.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
## piz-daint @ CSCS
Build instructions for the Piz Daint (GPU cluster).
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.