=========================================================
Run instructions for PRACE Accelerator Benchmark for GPAW
=========================================================

GPAW is a density-functional theory (DFT) program for ab initio electronic
structure calculations using the projector augmented wave method. It is
written mostly in Python and uses MPI for parallelisation.

GPAW is licensed under GPL and is freely available at:
  https://wiki.fysik.dtu.dk/gpaw/
  https://gitlab.com/gpaw/gpaw


Download benchmark
==================

The benchmark set is available at:
  https://github.com/mlouhivu/gpaw-benchmarks/tree/prace

or at the PRACE RI website (http://www.prace-ri.eu/ueabs/).


Small case: Carbon nanotube
===========================

A ground state calculation for a carbon nanotube in vacuum. By default uses a
6-6-10 nanotube with 240 atoms (freely adjustable) and serial LAPACK with an
option to use ScaLAPACK.

This benchmark is aimed at smaller systems, with an intended scaling range of
up to 10 nodes.

Input file: carbon-nanotube/input.py


Large case: Copper filament
===========================

A ground state calculation for a copper filament in vacuum. By default uses a
2x2x3 FCC lattice with 71 atoms (freely adjustable) and ScaLAPACK for
parallelisation.

This benchmark is aimed at larger systems, with an intended scaling range of
up to 100 nodes.

Input file: copper-filament/input.py


Running benchmarks
==================

No special command line options or environment variables are needed to run the
benchmarks on GPGPUs or KNL (Xeon Phi Knights Landing) MICs. One can simply
say e.g.
  mpirun -np 256 gpaw-python input.py


For KNCs (Xeon Phi Knights Corner), one needs to use a wrapper script to set
correct affinities for pyMIC (see setup/affinity-wrapper.sh for an example)
and to set two environment variables for GPAW:
  GPAW_OFFLOAD=1  (to turn on offloading)
  GPAW_PPN=<no. of MPI tasks per node>

For example, in a SLURM system, this could be:
  GPAW_PPN=12 GPAW_OFFLOAD=1 mpirun -np 256 -bootstrap slurm \
    ./affinity-wrapper.sh gpaw-python input.py


Example job scripts (setup/job-*.sh) for different accelerator architectures
are provided together with related machine specifications (setup/specs.*) that
may offer a helpful starting point (especially for KNCs).