Commit 8244f540 authored by Kurt Lust's avatar Kurt Lust
Browse files

Update of the GPAW documentation.

parent 6b9a0a55
......@@ -37,8 +37,8 @@ There is currently no active support for non-CUDA accelerator platforms.
For the UEABS benchmark version 2.2, the following versions of GPAW were tested:
* CPU-based:
* Version 1.5.3 as this one is the last of the 1.5 branch and since the GPU version
is derived from 1.5.2.
* Version 1.5.2 as this one is the last of the 1.5 branch and since the GPU version
is derived from this version.
* Version 20.1.0, the most recent version during the development of the UEABS
2.2 benchmark suite.
* GPU-based: There is no official release or version number. The UEABS 2.2 benchmark
......@@ -60,10 +60,18 @@ Input file: [benchmark/1_S_carbon-nanotube/input.py](benchmark/1_S_carbon-nanotu
### Case M: Copper filament
A ground state calculation for a copper filament in vacuum. By default uses a
2x2x3 FCC lattice with 71 atoms (freely adjustable) and ScaLAPACK for
parallelisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
3x4x4 FCC lattice with 71 atoms (freely adjustable through the variables `x`,
`y` and `z` in the input file) and ScaLAPACK for
parallellisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
Input file: [benchmark/2_M_carbon-nanotube/input.py](benchmark/2_M_copper-filament/input.py)
Input file: [benchmark/2_M_copper-filament/input.py](benchmark/2_M_copper-filament/input.py)
The benchmark was tested using 1000 and 1024 cores. For some core configurations, one may
get error messages similar to ``gpaw.grid_descriptor.BadGridError: Grid ... to small
for ... cores``. If one really wants to run the benchmark for those number of cores,
one needs to adapt the values of `x`, `y` and `z` in `input.py`. However, this
changes the benchmark so results cannot be compared easily with benchmark runs for
different values of these variables.
### Case L: Silicon cluster
......@@ -72,7 +80,7 @@ cluster has a radius of 15Å (freely adjustable) and consists of 702 atoms,
and ScaLAPACK is used for parallelisation. Expected to scale up to 1000 nodes
and/or 10000 MPI tasks.
Input file: [benchmark/3_L_carbon-nanotube/input.py](benchmark/3_L_silicon-cluster/input.py)
Input file: [benchmark/3_L_silicon-cluster/input.py](benchmark/3_L_silicon-cluster/input.py)
## Mechanics of building the benchmark
......@@ -82,18 +90,37 @@ last version with the old numbering. In 2019 the development team switched
to a version numbering scheme based on year, month and patchlevel, e.g.,
19.8.1 for the second version released in August 2019.
A further major change affecting both the build process and the mechanics of running
the benchmark happened in version 20.1.0. Versions up to and including 19.8.1 use a
wrapper executable `gpaw-python` that replaces the Python interpreter (it internally
links to the libpython library) and provides the MPI functionality. From version 20.1.0
the standard Python interpreter is used and the MPI functionality is included in the `_gpaw.so`
shared library, though there is still an option in the build process (not tested for
the UEABS benchmarks) to generate that wrapper instead.
Another change is in the Python packages used to install GPAW. Versions up to
and including 19.8.1 use the `distutils` package while versions 20.1.0 and later
are based on `setuptools`. This does affect the installation process.
GPAW for a while supports two different ways to run in parallel distributed memory mode:
* Using a wrapper executable `gpaw-python` that replaces the Python interpreter (it internally
links to the libpython library) and that provides the MPI functionality.
* Using the standard Python interpreter, including the MPI functionality in the
`_gpaw.so` shared library.
In the `distutils`-based versions, the wrapper script approach is the default behaviour,
while in the `setuptools`-based versions, the approach using the standard Python interpreter
is the preferred one in the manual. Even though the code in the `setuptools`-based
versions still includes the option to use the wrapper script approach, it does not
work in the tested version 20.1.0.
### Available instructions
The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) only contains the
[installation instructions](https://wiki.fysik.dtu.dk/gpaw/index.html) for the current version.
For the installation instructions with a list of dependencies for older versions,
download the code (see below) and look for the file `doc/install.rst` or go to the
[GPAW GitLab](https://gitlab.com/gpaw), select the tag for the desired version and
view the file `doc/install.rst`.
The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) also provides some
[platform specific examples](https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html).
### List of dependencies
GPAW is Python code (3.5 or newer) but it also contains some C code for some performance-critical
GPAW is Python code but it also contains some C code for some performance-critical
parts and to interface to a number of libraries on which it depends.
Hence GPAW has the following requirements:
......@@ -101,15 +128,17 @@ Hence GPAW has the following requirements:
* BLAS, LAPACK, BLACS and ScaLAPACK. ScaLAPACK is optional for GPAW, but mandatory
for the UEABS benchmarks. It is used by the medium and large cases and optional
for the small case.
* Python 3.5 or newer
* Python. GPAW 1.5.2 requires
Python 2.7 or 3.4-3.7, GPAW 19.8.1 requires 3.4-3.7 and GPAW 20.1.0 Python 3.5-3.8.
* Mandatory Python packages:
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 19.8.1/20.1.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 19.8.1/20.1.0)
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0)
* [FFTW](http://www.fftw.org) is highly recommended. As long as the optional libvdwxc
component is not used, the MKL FFTW wrappers can also be used. Recent versions of
GPAW can even show good performance using just the NumPy-provided FFT routines provided
GPAW also show good performance using just the NumPy-provided FFT routines provided
that NumPy has been built with a highly optimized FFT library.
* [LibXC](https://www.tddft.org/programs/libxc/) 3.X or 4.X. LibXC is a library
* [LibXC](https://www.tddft.org/programs/libxc/) 2.X or newer for GPAW 1.5.2,
3.X or 4.X for GPAW 19.8.1 and 20.1.0. LibXC is a library
of exchange-correlation functions for density-functional theory
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW
......@@ -128,9 +157,17 @@ Hence GPAW has the following requirements:
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
In addition, the GPU version needs:
* CUDA toolkit
* NVIDIA CUDA toolkit
* [PyCUDA](https://pypi.org/project/pycuda/)
Installing GPAW also requires a number of standard build tools on the system, including
* [GNU autoconf](https://www.gnu.org/software/autoconf/) is needed to generate the
configure script for libxc
* [GNU Libtool](https://www.gnu.org/software/libtool/) is needed. If not found,
the configure process of libxc produces very misleading
error messages that do not immediately point to libtool missing.
* [GNU make](https://www.gnu.org/software/make/)
### Download of GPAW
......@@ -155,21 +192,16 @@ git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
### Install
Official generic [installation instructions](https://wiki.fysik.dtu.dk/gpaw/install.html)
and
[platform specific examples](https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html)
are provided in the [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/).
Crucial for the configuration of GPAW is a proper `customize.py` (GPAW 19.8.1 and
earlier) or `siteconfig.py` (GPAW 20.1.0 and later) file. The defaults used by GPAW
may not offer optimal performance and the automatic detection of the libraries also
fails on some systems.
The UEABS repository contains additional instructions:
* [general instructions](installation.md) - Under development
* [general instructions](build/build-cpu.md) - Under development
* [GPGPUs](build/build-cuda.md) - To check
Example [build scripts](build/examples/) are also available for some PRACE
Example [build scripts](build/examples/) are also available for some PRACE and non-PRACE
systems.
......@@ -187,7 +219,9 @@ right from this repository.
### Running the benchmarks
#### Versions up to and including 19.8.1 of GPAW
#### Using the `gpaw-python` wrapper script
This is the default approach for versions up to and including 19.8.1 of GPAW
These versions of GPAW come with their own wrapper executable, `gpaw-python`,
to start a MPI-based GPAW run.
......@@ -199,7 +233,9 @@ properly with the resource manager. E.g., on Slurm systems, use
srun gpaw-python input.py
```
#### GPAW 20.1.0 (and likely later)
#### Using the regular Python interpreter and parallel GPAW shared library
This is the default method for GPAW 20.1.0 (and likely later).
The wrapper executable `gpaw-python` is no longer available in the default parallel
build of GPAW. There are now two different ways to start GPAW.
......@@ -229,8 +265,8 @@ would do.
Example [job scripts](scripts/) (`scripts/job-*.sh`) are provided for
different PRACE systems that may offer a helpful starting point.
TODO: Update the examples.
*TODO: Update the examples as testing on other systems goes on.*
## Verification of Results
......@@ -240,9 +276,9 @@ TODO.
### Case M: Copper filament
TODO.
TODO. Convergence problems.
### Case L: Silicon cluster
TODO.
TODO. Get the medium case to run before spending time on the large one.
# Detailed GPAW installation instructions on non-acclerated systems
These instructions are in addition to the brief instructions in [README.md](../README.md).
## Detailed dependency list
### Libraries and Python interpreter
GPAW needs (for the UEABS benchmarks)
* [Python](https://www.python.org/): GPAW 1.5.2 supports Python 2.7 and 3.4-3.7.
GPAW 19.8.1 needs Python 3.4-3.7 and GPAW 20.1.0 requires Python 3.5-3.8.
* [MPI library](https://www.mpi-forum.org/)
* [LibXC](https://www.tddft.org/programs/libxc/). GPAW 1.5.2 requires LibXC 1.5.2
or later. GPAW 19.8.1 and 20.1.0 need LibXC 3.x or 4.x.
* (Optimized) [BLAS](http://www.netlib.org/blas/) and
[LAPACK](http://www.netlib.org/lapack/) libraries.
There are both commercial and free and open source versions of these libraries.
Using the [reference implementation of BLAS from netlib](http://www.netlib.org/blas/)
will give very poor performance. Most optimized LAPACK libraries actually only
optimize a few critical routines while the remaining routines are compiled from
the reference version. Most processor vendors for HPC machines and system vendors
offer optmized versions of these libraries.
* [ScaLAPACK](http://www.netlib.org/scalapack/) and the underlying communication
layer [BLACS](http://www.netlib.org/blacs/).
* [FFTW](http://www.fftw.org/) or compatible FFT library.
For the UEABS benchmarks, the double precision, non-MPI version is sufficient.
GPAW also works with the
[Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html)
FFT routines when using the FFTW wrappers provided with that product.
For the GPU version, the following packages are needed in addition to the packages
above:
* CUDA toolkit
* [PyCUDA](https://pypi.org/project/pycuda/)
Optional components of GPAW that are not used by the UEABS benchmarks:
* [libvdwxc](https://gitlab.com/libvdwxc/libvdwxc), a portable C library
of density functionals with van der Waals interactions for density functional theory.
This library does not work with the MKL FFTW wrappers as it needs the MPI version
of the FFTW libraries too.
* [ELPA](https://elpa.mpcdf.mpg.de/),
which should improve performance for large systems when GPAW is used in
[LCAO mode](https://wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html)
### Python packages
GPAW needs
* [wheel](https://pypi.org/project/wheel/) is needed in most (if not all) ways of
installing the packages from source.
* [NumPy](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0)
* Installing NumPy from source will also require
[Cython](https://pypi.org/project/Cython/)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0)
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW. Required versions are 3.17.0 or later for
GPAW 1.5.2 and 3.18.0 or later for GPAW 19.8.1 or 20.1.0.
ASE has a couple of dependendencies
that are not needed for running the UEABS benchmarks. However, several Python
package install methods will trigger the installation of those packages, and
with them may require a chain of system libraries.
* ASE does need NumPy and SciPy, but these are needed anyway for GPAW.
* [matplotlib](https://pypi.org/project/matplotlib/), at least version 2.0.0.
This package is optional and not really needed to run the benchmarks.
Matplotlib pulls in a lot of other dependencies. When installing ASE with pip,
it will try to pull in matplotlib and its dependencies
* [pillow](https://pypi.org/project/Pillow/) needs several exgternal
libraries. During the development of the benchmarks, we needed at least
zlib, libjpeg-turbo (or compatible libjpeg library) and freetype. Even
though the pillow documentation claimed that libjpeg was optional,
it refused to install without.
* [kiwisolver](https://pypi.org/project/kiwisolver/): Contains C++-code
* [pyparsing](https://pypi.org/project/pyparsing/)
* [Cycler](https://pypi.org/project/Cycler/), which requires
* [six](https://pypi.org/project/six/)
* [python-dateutil](https://pypi.org/project/python-dateutil/), which also
requires
* [six](https://pypi.org/project/six/)
* [Flask](https://pypi.org/project/Flask/) is an optional dependency of ASE
that is not automatically pulled in by `pip` in versions of ASE tested during
the development of this version of the UEABS. It has a number of dependencies
too:
* [Jinja2](https://pypi.org/project/Jinja2/)
* [MarkupSafe](https://pypi.org/project/MarkupSafe/), contains some C
code
* [itsdangerous](https://pypi.org/project/itsdangerous/)
* [Werkzeug](https://pypi.org/project/Werkzeug/)
* [click]()
## Tested configurations
* Python
* Libraries used during the installation of Python:
* ncurses 6.2
* libreadline 8.0
* libffi 3.3
* zlib 1.2.11
* OpenSSL 1.1.1g, but only because EasyBuild was used and requires it.
* Python will of course pick up several other libraries that it might find on
the system. The benchmark installation was tested on a system with very few
development packages of libraries installed in the system image. Tcl/Tk
and SQLite3 development packages in particular where not installed, so the
standard Python library packages sqlite3 and tkinter were not fully functional.
* Python packages
* wheel
* Cython
* NumPy
* SciPy
* ASE
* GPAW
The table below give the combinations of major packages Python, NumPy, SciPy, ASE and
GPAW that were tested:
| Python | NumPy | SciPy | ASE | GPAW |
|:-------|:-------|:------|:-------|:--------|
| 3.7.9 | 1.18.5 | 1.4.1 | 3.17.0 | 1.5.2 |
| 3.7.9 | 1.18.5 | 1.4.1 | 3.18.0 | 19.8.1 |
| 3.8.5 | 1.18.5 | 1.4.1 | 3.19.1 | 20.1.0 |
## Installing all prerequisites
We do not include the optimized mathematical libraries in the instructions (BLAS, LAPACK,
FFT library, ...). Also, the instructions below will need to be adapted to the specific
libraries that are being used.
Other prerequisites:
* libxc
* Python interpreter
* Python package NumPy
* Python package SciPy
* Python package ase
### Installing libxc
* Installing libxc requires GNU automake and GNU buildtool besides GNU make and a
C compiler. The build process is the usual GNU configure - make - make install
cycle, but the `configure` script still needs to be generated with autoreconf.
### Installing Python from scratch
The easiest way to get Python on your system is to download an existing distribution
(one will likely already be installed on your system). Python itself does have a lot
of dependencies though, definitely in its Standard Python Library. Many of the
standard packages are never needed when executing the benchmark cases. Isolating them
to compile a Python with minimal dependencies is beyond the scope though. We did
compile Python without the necessary libraries for the standard libraries sqlite3
and tkinter (the latter needing Tcl/Tk).
Even though GPAW contains a lot of Python code, the Python interpreter is not the main
performance-determining factor in the GPAW benchmark. Having a properly optimized installation
of NumPy, SciPy and GPAW itself proves much more important.
## Configuring and installing GPAW
### GPAW 1.5.2
* GPAW 1.5.2 uses `distutils`. Customization of the installation process is possible
through the `customize.py` file.
* The FFT library: According to the documentation, the following strategy is used
* The compile process searches (in this order) for ``libmkl_rt.so``,
``libmkl_intel_lp64.so`` and ``libfftw3.so`. First one found will be
loaded.
* If none is found, the built-in FFT from NumPy will be used. This does not need
to be a problem if NumPy provides a properly optimized FFT library.
* The choice can also be overwritten using the GPAW_FFTWSO environment variable.
### GPAW 19.8.1
* GPAW 19.8.1 uses `distutils`. Customization of the installation process is possible
through a `customize.py` file.
* The selection process of the FFT library has changed from version 1.5.2. It is
now possible to specify the FFT library in `customize.py` or to simply select to
use the NumPy FFT routines.
### GPAW 30.1.0
* GPAW 20.1.0 uses `setuptools`. Customization of the installation process is possible
through the `siteconfig.py` file.
* The selection process of the FFT library is the same as in version 19.8.1, except
that the settings are now in `siteconfrig.py` rather than `customize.py`.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment