Commit 6bd66b98 authored by Martti Louhivuori's avatar Martti Louhivuori
Browse files

Update CUDA instructions

parent f26b45e4
GPGPUs
======
GPAW has a separate CUDA version available for Nvidia GPGPUs. Nvidia has
released multiple versions of the CUDA toolkit. In this work just CUDA 7.5 was
tested with Tesla K20, Tesla K40 and Tesla K80 cards.
GPAW has a separate CUDA version available for NVIDIA GPGPUs. Source code is
available in GPAW's repository as a separate branch called 'cuda'.
An up-to-date development version is currently available at
[https://gitlab.com/mlouhivu/gpaw](https://gitlab.com/mlouhivu/gpaw).
Source code is available in GPAW's repository as a separate branch called
'cuda'. To obtain the code, use e.g. the following commands:
To obtain the latest version of the code, use e.g. the following command:
```
git clone https://gitlab.com/gpaw/gpaw.git
cd gpaw
git checkout cuda
git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
```
or download it from: https://gitlab.com/gpaw/gpaw/tree/cuda
Alternatively, the source code is also available with minor modifications
required to work with newer versions of CUDA and Libxc (incl. example
installation settings) at:
https://gitlab.com/atekin/gpaw-cuda.git
Patches needed to work with newer versions of CUDA and example setup scripts
for GPAW using dynamic links to Libxc are available also separately at:
https://github.com/mlouhivu/gpaw-cuda-patches.git
or download it from: https://gitlab.com/mlouhivu/gpaw/tree/cuda
Software requirements
---------------------
CUDA branch of the GPAW is based on version 0.9.1.13528 and has similar
software requirements to the main branch. To compile the code, one needs to
use Intel compile environment.
For example, the following versions are known to work:
* Intel compile environment with Intel MKL and Intel MPI (2015 update 3)
* Python (2.7.11)
* ASE (3.9.1)
* Libxc (3.0.0)
* CUDA (7.5)
* HDF5 (1.8.16)
* Pycuda (2016.1.2)
In addition to the normal software requirements of GPAW, the GPU version
requires the [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) and the
[PyCUDA](https://pypi.org/project/pycuda/) python module.
Install instructions
--------------------
Before installing the CUDA version of GPAW, the required packages should be
compiled using Intel compilers. In addition to using Intel compilers, there
are two additional steps compared to a standard installation:
The only additional step to installing GPAW is that in the CUDA version one
needs to build the CUDA kernels before building the rest of the GPAW. This is
done in the `c/cuda/` directory that contains the CUDA kernels. There is a
customisable Makefile (`make.inc`) that can be edited before running the
`make` command.
1. Compile the CUDA files after preparing a suitable `make.inc` (in `c/cuda/`)
by modifying the default options and paths to match your system. The
following compiler options may offer a good starting point.
So, the steps to install the CUDA version of GPAW are:
```shell
CC = icc
CCFLAGS = $(CUGPAW_DEFS) -fPIC -std=c99 -m64 -O3
NVCC = nvcc -ccbin=icpc
NVCCFLAGS = $(CUGPAW_DEFS) -O3 -arch=sm_20 -m64 --compiler-options '-fPIC -O3'
```
1. Edit the Makefile for the CUDA kernels (`c/cuda/make.inc`).
To use a dynamic link to Libxc, please add a corresponding include flag to
the CUGPAW_INCLUDES (e.g. `-I/path/to/libxc/include`) and a
Modify the default options and paths to match your system. The essential
parts are the include paths for libraries (MPI, CUDA) and the build options
for `nvcc` to target the correct GPU architecture.
It is possible that you may also need to add additional include flags for
MKL and Libxc in CUGPAW_INCLUDES (e.g. `-I/path/to/mkl/include`).
2. Build the CUDA kernels.
After making the necessary changes, simply run make (in the `c/cuda`
path).
```bash
cd c/cuda
make
cd -
```
2. Edit your GPAW setup script (`customize.py`) to add correct link and
compile options for CUDA. The relevant lines are e.g.:
3. Edit the GPAW setup script (`customize.py`).
Add correct link and compile options for CUDA (and possibly other
libraries). The relevant lines for CUDA are e.g.:
```python
define_macros += [('GPAW_CUDA', '1')]
libraries += [
'gpaw-cuda',
'cublas',
'cudart',
'stdc++'
]
libraries += ['gpaw-cuda', 'cublas', 'cudart', 'stdc++']
library_dirs += [
'./c/cuda',
'/path/to/cuda/lib64'
......@@ -88,3 +62,19 @@ are two additional steps compared to a standard installation:
]
```
4. Apply patch to an eigensolver (only for code based on version 1.1.0).
Cherry pick an upstream patch to remove obsolete code from the DIIS step
in `gpaw/eigensolvers/rmm_diis.py` that is still in version 1.1.0 but has
not been ported to GPUs. The patch is available in
[build/patch-rmmdiis.diff](patch-rmmdiis.diff).
```bash
patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
```
5. Build and install GPAW.
```bash
python setup.py install --prefix=$TARGET_DIRECTORY
```
commit 761cba649d58e2d2f24c0a1e2fdad917b5929679
Author: Martti Louhivuori <martti.louhivuori@csc.fi>
Date: Thu May 18 10:56:07 2017 +0300
Remove obsolete error calculation from DIIS step
diff --git a/gpaw/eigensolvers/rmmdiis.py b/gpaw/eigensolvers/rmmdiis.py
index 7d60553..d182713 100644
--- a/gpaw/eigensolvers/rmmdiis.py
+++ b/gpaw/eigensolvers/rmmdiis.py
@@ -299,23 +299,6 @@ class RMMDIIS(Eigensolver):
P_axi, kpt.eps_n[n_x], R_xG, n_x,
calculate_change=True)
self.timer.stop('Calculate residuals')
- self.timer.start('Calculate errors')
- errors_new_x = np.zeros(B)
- # errors_x[:] = 0.0
- for n in range(n1, n2):
- if kpt.f_n is None:
- weight = kpt.weight
- else:
- weight = kpt.f_n[n]
- if self.nbands_converge != 'occupied':
- if wfs.bd.global_index(n) < self.nbands_converge:
- weight = kpt.weight
- else:
- weight = 0.0
- errors_new_x[n-n1] += weight * integrate(R_xG[n - n1],
- R_xG[n - n1])
- comm.sum(errors_x)
- self.timer.stop('Calculate errors')
self.timer.stop('DIIS step')
# Final trial step
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment