Commit baa5d752 authored by Andrew Emerson's avatar Andrew Emerson
Browse files

Updated README.md for Quantum Espresso

parent b2546dfe
......@@ -9,14 +9,12 @@ Full documentation is available from the project website [QuantumEspresso](https
In this README we give information relevant for its use in the UEABS.
### Standard CPU version
For the UEABS activity we have used mainly version v6.3 but later versions are now available.
For the UEABS activity we have used mainly version v6.5 but later versions are now available.
### GPU version
The GPU port of Quantum Espresso is a version of the program which has been
completely re-written in CUDA FORTRAN by Filippo Spiga (NVidia). The version program used in these
experiments is v6.0, even though further versions becamse available later during the
activity.
completely re-written in CUDA FORTRAN. The version program used in these
experiments is v6.5a1, even though later versions may be available.
## Installation and requirements
### Standard
......@@ -24,7 +22,7 @@ The Quantum Espresso source can be downloaded from the projects GitHub repositor
### GPU version
For complete build requirements and information see the following GitHub site:
[QE-GPU](https://github.com/fspiga/qe-gpu)
[QE-GPU](https://gitlab.com/QEF/q-e-gpu/-/releases)
A short summary is given below:
Essential
......@@ -45,14 +43,14 @@ Optional
### Standard
From the website, for example:
```bash
wget https://github.com/QEF/q-e/releases/download/qe-6.3/qe-6.3.tar.gz
wget https://github.com/QEF/q-e/releases/download/qe-6.5/qe-6.5.tar.gz
```
### GPU
Available from the web site given above. You can use, for example, ```git clone```
Available from the web site given above. You can use, for example, ```wget```
to download the software:
```bash
git clone https://github.com/fspiga/qe-gpu.git
wget https://gitlab.com/QEF/q-e-gpu/-/archive/qe-gpu-6.5a1/q-e-gpu-qe-gpu-6.5a1.tar.gz
```
## Compiling and installing the application
......@@ -71,26 +69,36 @@ make; make install
```
### GPU
Check the __README.md__ file in the downloaded files since the
procedure varies from distribution to distribution.
Most distributions do not have a ```configure``` command. Instead you copy a __make.inc__
file from the __install__ directory, and modify that directly before running make.
A number of templates are available in the distribution:
- make.inc_x86-64
- make.inc_CRAY_PizDaint
- make.inc_POWER_DAVIDE
- make.inc_POWER_SUMMITDEV
The second and third are particularly relevant in the PRACE infrastructure (ie. for CSCS
PizDaint and CINECA DAVIDE).
Run __make__ to see the options available. For the UEABS you should select the
pw program (the only module currently available)
The GPU version is configured similarly to the CPU version, the only exception being that the configure script
will check for the presence of PGI and CUDA libraries.
A typical configure might be
```bash
./configure --with-cuda=XX --with-cuda-runtime=YY --with-cuda-cc=ZZ --enable-openmp [ --with-scalapack=no ]
```
where `XX` is the location of the CUDA Toolkit (in HPC environments is
generally `$CUDA_HOME`), `YY` is the version of the cuda toolkit and `ZZ`
is the compute capability of the card.
For example,
```bash
./configure --with-cuda=$CUDA_HOME --with-cuda-cc=60 --with-cuda-runtime=9.2
```
The __dev-tools/get_device_props.py__ script is available if you dont know these values.
Compilation is then performed as normal by
```
make pw
```
The QE-GPU executable will appear in the directory `GPU/PW` and is called `pw-gpu.x`.
#### Example compilation of Quantum Espresso for GPU based machines
```bash
module load pgi cuda
./configure --with-cuda=$CUDA_HOME --with-cuda-cc=70 --with-cuda-runtime=10.2
make -j8 pw
```
## Running the program - general procedure
......@@ -103,7 +111,7 @@ input files are of two types:
The pseudopotential files are placed in a directory specified in the
control file with the tag pseudo\_dir. Thus if we have
```shell
```bash
pseudo_dir=./
```
then QE-GPU will look for the pseudopotential
......@@ -111,13 +119,13 @@ files in the current directory.
If using the PRACE benchmark suite the data files can be
downloaded from the PRACE respository. For example,
```shell
```bash
wget https://repository.prace-ri.eu/ueabs/Quantum_Espresso/QuantumEspresso_TestCaseA.tar.gz
```
Once uncompressed you can then run the program like this (e.g. using
MPI over 16 cores):
```shell
```bash
mpirun -n 16 pw-gpu.x -input pw.in
```
......@@ -126,6 +134,22 @@ but check your system documentation since mpirun may be replaced by
allowed to run MPI programs interactively without using the
batch system.
### Running on GPUs
The procedure is identical to running on non accelerator-based hardware.
If GPUs are being used then the following will appear in the program output:
```
GPU acceleration is ACTIVE.
```
GPU acceleration can be switched off by setting the following environment variable:
```bash
$ export USEGPU=no
```
### Parallelisation options
Quantum Espresso uses various levels of parallelisation, the most important being MPI parallelisation
over the *k points* available in the input system. This is achieved with the ```-npool``` program option.
......@@ -154,14 +178,11 @@ srun -n 64 pw.x -npool 2 -ndiag 4 -input pw.in
### Hints for running the GPU version
#### Memory limitations
The GPU port of Quantum Espresso runs almost entirely in the GPU memory. This means that jobs are restricted
by the memory of the GPU device, normally 16-32 GB, regardless of the main node memory. Thus, unless many nodes are used the user is likely to see job failures due to lack of memory, even for small datasets.
For example, on the CSCS Piz Daint supercomputer each node has only 1 NVIDIA Tesla P100 (16GB) which means that you will need at least 4 nodes to run even the smallest dataset (AUSURF in the UEABS).
## Execution
In the UEABS repository you will find a directory for each computer system tested, together with installation
......@@ -173,14 +194,14 @@ In the following we describe in detail the execution procedure for the Marconi
Quantum Espresso has already been installed on the cluster
and can be accessed via a specific module:
``` shell
``` bash
module load profile/phys
module load autoload qe/6.5
```
An example SLURM batch script for the A2 partition is given below:
An example SLURM batch script is given below:
``` shell
``` bash
#!/bin/bash
#SBATCH --time=06:00:00 # Walltime in hh:mm:ss
#SBATCH --nodes=4 # Number of nodes
......@@ -199,32 +220,26 @@ module load autoload qe/6.5
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MKL_NUM_THREADS=${OMP_NUM_THREADS}
mpirun pw.x -npool 4 -input file.in > file.out
srun pw.x -npool 4 -input file.in > file.out
```
In the above with the SLURM directives we have asked for 2 KNL nodes (each with 68 cores) in
cache/quadrant mode and 93 Gb main memory each. We are running QE in MPI-only
mode using 18 MPI processes/node with the k-points in 4 pools.
In the above with the SLURM directives we have asked for 4 nodes, 18 MPI tasks per node and 2 OpenMP threads
per task.
Note that this script needs to be submitted using the KNL scheduler as follows:
Note that this script needs to be submitted using SLURM scheduler as follows:
``` shell
module load env-knl
``` bash
sbatch myjob
```
Please check the Cineca documentation for information on using the
[Marconi KNL partition]
(https://wiki.u-gov.it/confluence/display/SCAIUS/UG3.1%3A+MARCONI+UserGuide#UG3.1:MARCONIUserGuide-SystemArchitecture).
## UEABS test cases
| UEABS name | QE name | Description | k-points | Notes|
|------------|---------------|-------------|----------|------|
| Small test case | AUSURF | 112 atoms | 2 | < 4-8 nodes on most systems |
| Large test case | TA2O5 | Tantalum oxide| 26| Medium scaling, often 20 nodes |
| Very Large test case | CNT | Carbon nanotube | | Large scaling runs only. Memory and time very requirements high|
| Large test case | GRIR443 | 432 | 4| Medium scaling, often 20 nodes |
| Very Large test case | CNT | Carbon nanotube | | Large scaling runs only. Memory and time requirements very high|
__Last updated: 29-April-2019__
__Last updated: 22-October-2020__
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment