Commit 0d222709 authored by Kurt Lust's avatar Kurt Lust
Browse files

Updates based on the review and recent experiences at LRZ.

parent 5b7f210c
......@@ -116,7 +116,7 @@ interpreter is used and the MPI functionality is included in the `_gpaw.so` shar
### Available instructions
The [GPAW wiki](https://wiki.fysik.dtu.dk/gpaw/) only contains the
[installation instructions](https://wiki.fysik.dtu.dk/gpaw/index.html) for the current version.
[installation instructions](https://wiki.fysik.dtu.dk/gpaw/install.html) for the current version.
For the installation instructions with a list of dependencies for older versions,
download the code (see below) and look for the file `doc/install.rst` or go to the
[GPAW GitLab](https://gitlab.com/gpaw), select the tag for the desired version and
......@@ -205,7 +205,7 @@ may not offer optimal performance and the automatic detection of the libraries a
fails on some systems.
The UEABS repository contains additional instructions:
* [general instructions](build/build-cpu.md)
* [general instructions](build/build-CPU.md)
Example [build scripts](build/examples/) are also available.
......@@ -297,11 +297,17 @@ on one hand and version 21.1.0 on the other hand.
The expected values are:
* Number of iterations: Between 30 and 35
* Dipole (3rd component):
* 20.1.0 and 20.10.0: between -0.493 and -0.491
* 21.1.0: between -0.462 and -0.461
* Fermi level:
* 20.1.0 and 20.10.0: between -2.67 and -2.66
* 21.1.0: between -2.59 and -2.58
* Dipole (3rd component):between -0.493 and -0.491
* Fermi level:between -2.67 and -2.66
* Extrapolated energy: Between -3784 and -3783
Note: Though not used for the benchmarking in the final report, some testing was done
with version 21.1.0 also. In this version, some external library routines were replaced
by new internal implementations that cause changes in some results. For 21.1.0, the
expected values are:
* Number of iterations: Between 30 and 35
* Dipole (3rd component): between -0.462 and -0.461
* Fermi level: between -2.59 and -2.58
* Extrapolated energy: Between -3784 and -3783
......@@ -126,6 +126,14 @@ GPAW that were tested:
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.4 | 1.19.5 | 1.5.4 |
Note: On some systems compiling SciPy 1.5.4 with NumPy 1.19.5 produced errors. On those
systems NumPy 1.18.5 was used.
Other configurations that were only tested on a limited number of clusters:
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 21.1.0 | 3.21.1 | 3.9.4 | 1.19.5 | 1.5.4 |
......@@ -162,8 +170,6 @@ Other prerequisites:
* Untar the file in the build directory.
### Installing Python from scratch
The easiest way to get Python on your system is to download an existing distribution
......
GPGPUs
======
**These instructions were not updated for UEABS release 2.2 as the GPU version
was in full redevelopment at that time to also work for the AMD GPUs used in the
LUMI pre-exascale system. They may or may not be useful anymore for the new
versions once they are finished.**
GPAW has a separate CUDA version available for NVIDIA GPGPUs. Source code is
available in GPAW's repository as a separate branch called 'cuda'.
An up-to-date development version is currently available at
......
Xeon Phi MICs
=============
Intel's MIC architecture includes two distinct generations of processors:
1st generation Knights Corner (KNC) and 2nd generation Knights Landing (KNL).
KNCs require a specific offload version of GPAW, whereas KNLs use standard
GPAW.
KNC (Knights Corner)
--------------------
For KNCs, GPAW has adopted an offload-to-the-MIC-coprocessor approach similar
to GPGPUs. The offload version of GPAW uses the stream-based offload module
pyMIC (https://github.com/01org/pyMIC) to offload computationally intensive
matrix calculations to the MIC co-processors.
Source code is available in GPAW's repository as a separate branch called
'mic'. To obtain the code, use e.g. the following commands:
```
git clone https://gitlab.com/gpaw/gpaw.git
cd gpaw
git checkout mic
```
or download it from: https://gitlab.com/gpaw/gpaw/tree/mic
A ready-to-use install package with examples and instructions is also
available at:
https://github.com/mlouhivu/gpaw-mic-install-pack.git
### Software requirements
The offload version of GPAW is roughly equivalent to the 0.11.0 version of
GPAW and thus has similar requirements (for software and versions).
For example, the following versions are known to work:
* Python (2.7.x)
* ASE (3.9.1)
* NumPy (1.9.2)
* Libxc (2.1.x)
In addition, pyMIC requires:
* Intel compile environment with Intel MKL and Intel MPI
* Intel MPSS (Manycore Platform Software Stack)
### Install instructions
In addition to using Intel compilers, there are three additional steps apart
from standard installation:
1. Compile and install Numpy with a suitable `site.cfg` to use MKL, e.g.
```
[mkl]
library_dirs = /path/to/mkl/lib/intel64
include_dirs = /path/to/mkl/include
lapack_libs =
mkl_libs = mkl_rt
```
2. Compile and install [pyMIC](https://github.com/01org/pyMIC) before GPAW.
3. Edit your GPAW setup script (`customize.py`) to add correct link and
compile options for offloading. The relevant lines are e.g.:
```python
# offload to KNC
extra_compile_args += ['-qoffload-option,mic,compiler,"-qopenmp"']
extra_compile_args += ['-qopt-report-phase=offload']
# linker settings for MKL on KNC
mic_mkl_lib = '/path/to/mkl/lib/mic/'
extra_link_args += ['-offload-option,mic,link,"-L' + mic_mkl_lib \
+ ' -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread"']
```
KNL (Knights Landing)
---------------------
For KNLs, one can use the standard version of GPAW, instead of the offload
version used for KNCs. Please refer to the generic installation instructions
for GPAW.
### Software requirements
- https://wiki.fysik.dtu.dk/gpaw/install.html
### Install instructions
- https://wiki.fysik.dtu.dk/gpaw/install.html
- https://wiki.fysik.dtu.dk/gpaw/platforms/platforms.html
It is advisable to use Intel compile environment with Intel MKL and Intel MPI
to take advantage of their KNL optimisations. To enable the AVX512 vector
sets supported by KNLs, one needs to use the compiler option `-xMIC-AVX512`
when installing GPAW.
To improve performance, one may also link to Intel TBB to benefit from an
optimised memory allocator (tbbmalloc). This can be done during installation
or at run-time by setting environment variable LD_PRELOAD to point to the
correct libraries, i.e. for example:
```bash
export LD_PRELOAD=$TBBROOT/lib/intel64/gcc4.7/libtbbmalloc_proxy.so.2
export LD_PRELOAD=$LD_PRELOAD:$TBBROOT/lib/intel64/gcc4.7/libtbbmalloc.so.2
```
It may also be beneficial to use hugepages together with tbbmalloc
(`export TBB_MALLOC_USE_HUGE_PAGES=1`).
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment