GPGPUs
======

**These instructions were not updated for UEABS release 2.2 as the GPU version
was in full redevelopment at that time to also work for the AMD GPUs used in the
LUMI pre-exascale system. They may or may not be useful anymore for the new
versions once they are finished.**

GPAW has a separate CUDA version available for NVIDIA GPGPUs. Source code is
available in GPAW's repository as a separate branch called 'cuda'.
An up-to-date development version is currently available at
[https://gitlab.com/mlouhivu/gpaw](https://gitlab.com/mlouhivu/gpaw).

To obtain the latest version of the code, use e.g. the following command:
```
git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
```
or download it from: https://gitlab.com/mlouhivu/gpaw/tree/cuda


Software requirements
---------------------

In addition to the normal software requirements of GPAW, the GPU version
requires the [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) and the
[PyCUDA](https://pypi.org/project/pycuda/) python module.


Install instructions
--------------------

The only additional step to installing GPAW is that in the CUDA version one
needs to build the CUDA kernels before building the rest of the GPAW. This is
done in the `c/cuda/` directory that contains the CUDA kernels. There is a
customisable Makefile (`make.inc`) that can be edited before running the
`make` command.

So, the steps to install the CUDA version of GPAW are:

1. Edit the Makefile for the CUDA kernels (`c/cuda/make.inc`).

   Modify the default options and paths to match your system. The essential
   parts are the include paths for libraries (MPI, CUDA) and the build options
   for `nvcc` to target the correct GPU architecture.

2. Build the CUDA kernels.

   ```bash
   cd c/cuda
   make
   cd -
   ```

3. Edit the GPAW setup script (`customize.py`).

   Add correct link and compile options for CUDA (and possibly other
   libraries). The relevant lines for CUDA are e.g.:
   ```python
   define_macros += [('GPAW_CUDA', '1')]
   libraries += ['gpaw-cuda', 'cublas', 'cudart', 'stdc++']
   library_dirs += [
           './c/cuda',
           '/path/to/cuda/lib64'
   ]
   include_dirs += [
           '/path/to/cuda/include'
   ]
   ```

4. Apply patch to an eigensolver (only for code based on version 1.1.0).

   Cherry pick an upstream patch to remove obsolete code from the DIIS step
   in `gpaw/eigensolvers/rmm_diis.py` that is still in version 1.1.0 but has
   not been ported to GPUs. The patch is available in
   [build/patch-rmmdiis.diff](patch-rmmdiis.diff).

   ```bash
   patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
   ```

5. Build and install GPAW.

   ```bash
   python setup.py install --prefix=$TARGET_DIRECTORY
   ```