README_ACC.md

# CP2K 

This guide to building CP2K and running the accelerator benchmarks is a work in progress and under review.

## Get the source

Obtain the code using one of the mechanisms described at <https://www.cp2k.org/download>, for example by checking out a release branch from the source code repository:

```
svn checkout http://svn.code.sf.net/p/cp2k/code/branches/cp2k-4_1-branch cp2k
```


## Requirements

Building CP2K to run the accelerator benchmarks requires Fortran and C compilers and a CUDA compiler (in case of a GPU platform), an MPI library, and the linear algebra libraries BLAS, LAPACK and ScaLAPACK. Some version of all of these is usually already present on relevant systems. The linear algebra libraries can be found as part of a single vendor-provided numerical library such as Intel's MKL or Cray's LibSci. The FFTW library - usually already installed - is recommended but not required.

The LIBINT library described below is required and you should install it before building CP2K. LIBGRID and LIBXSMM are recommended for improved performance.

### LIBINT

CP2K has been tested with version 1.1.4 of LIBINT. 

Obtain the source:

```
wget https://www.cp2k.org/static/downloads/libint-1.1.4.tar.gz
```

If you can, it is easiest to build LIBINT on the same processor architecture on which you will run CP2K. This typically correspond to being to compile directly on the relevant compute nodes of the machine. If this is not possible and if you are forced instead to compile on nodes with a different processor architecture to the compute nodes on which CP2K will eventually run, see the section below on cross-compiling LIBINT. 

More information about LIBINT can be found inside the CP2K distribution base directory in

```
/tools/hfx_tools/libint_tools/README_LIBINT
```


#### Compiling LIBINT on compute nodes
Uncompress libint-1.1.4.tar.gz, enter the directory libint-1.1.4, and execute the commands listed below for the relevant choice of compilers. 

##### Using GCC (not recommended for Xeon Phi KNL)

```
CC=gcc  
CXX=g++
CFLAGS="-O2 -ftree-vectorize -g -fno-omit-frame-pointer -march=native -ffast-math -fsanitize=thread"
CXXFLAGS=$CFLAGS

./configure --prefix=/path/to/desired/install/location --with-cc-optflags=$CFLAGS --with-cxx-optflags=$CXXFLAGS --with-libint-max-am=5 --with-libderiv-max-am1=4

make -j number_of_cores_available_to_you

make install
```

##### Using Intel compilers
```
CC=icc
CXX=icpc
CFLAGS=-O2
```

If compiling for the Xeon Phi KNL:

```
CFLAGS+=" -xMIC-AVX512"
```

Then (for all processor architectures)

```
CXXFLAGS=$CFLAGS  

./configure --prefix=/path/to/desired/install/location --with-cc-optflags=$CFLAGS --with-cxx-optflags=$CXXFLAGS --with-libint-max-am=5 --with-libderiv-max-am1=4

make -j number_of_cores_available_to_you

make install
```

#### Cross-compiling LIBINT for compute nodes
If you are forced to cross-compile LIBINT for compute nodes on nodes that have a different processor architecture, follow these instructions. They assume you will be able to call a parallel application launcher like ``mpirun`` or ``mpiexec`` during your build process in order to run compiled code.

Uncompress ``libint-1.1.4.tar.gz`` and enter the directory ``libint-1.1.4``.

In ``/src/lib/`` edit the files ``MakeRules`` and ``MakeRules.in``.  
On the last line of each file, replace

```
cd $(LIBSRCLINK); $(TOPOBJDIR)/src/bin/$(NAME)/$(COMPILER)
```
with

```
cd $(LIBSRCLINK); $(PARALLEL_LAUNCHER_COMMAND) $(TOPOBJDIR)/src/bin/$(NAME)/$(COMPILER)
```

Then run

```
export PARALLEL_LAUNCHER_COMMAND="mpirun -n 1"
```
replacing ``mpirun`` with a different parallel application launcher such as ``mpiexec`` or ``aprun`` if applicable. Now proceed with the instructions for either GCC or Intel compilers given below.

##### Using GCC (not recommended for Xeon Phi KNL)

```
CC=gcc  
CXX=g++
CFLAGS="-O2 -ftree-vectorize -g -fno-omit-frame-pointer -march=haswell -ffast-math -fsanitize=thread"
CXXFLAGS=$CFLAGS
```
Note that the above example is for cross-compiling for compute nodes carrying Intel Haswell processors. You should modify the ``-march`` flag as needed, choosing the canonical processor architecture name appropriate for compute nodes on your machine.

``` 
./configure cross_compiling=yes --prefix=/path/to/desired/install/location --with-cc-optflags=$CFLAGS --with-cxx-optflags=$CXXFLAGS --with-libint-max-am=5 --with-libderiv-max-am1=4

make -j number_of_cores_available_to_you

make install
```

##### Using Intel compilers

```
CC=icc
CXX=icpc
CFLAGS=-O2
```

If compiling for the Xeon Phi KNL:

```
CFLAGS+=" -xMIC-AVX512"
```

Then (for all processor architectures)

```
CXXFLAGS=$CFLAGS  

./configure --prefix=/path/to/desired/install/location --with-cc-optflags=$CFLAGS --with-cxx-optflags=$CXXFLAGS --with-libint-max-am=5 --with-libderiv-max-am1=4

make -j number_of_cores_available_to_you

make install
```





## Building CP2K 

The general procedure is to create a so-called arch file specifying the build parameters inside the ``arch`` directory in the CP2K distribution. Building the hybrid MPI + OpenMP ("psmp") version of CP2K in accordance with a given arch file is then accomplished by entering the ``makefiles`` directory in the distribution and running

```
make -j number_of_cores_available_to_you ARCH=arch_file_name VERSION=psmp
```

Detailed information about arch file and library options and overall build procedure can be found in the ``INSTALL`` readme file in the CP2K distribution base directory and by examining the many template arch files for different architectures and compiler choices in the ``arch`` directory. 

If the build is successful, the resulting executable ``cp2k.psmp`` can be found inside ``/exe/arch_file_name/`` in the CP2K base directory.


### CP2K arch file for Xeon Phi KNL

##### Using Intel compilers
An example arch file for Xeon Phi KNL using version 17 of the Intel compiler suite and linking to MKL and to FFTW is shown below. Linking variables for MKL are set in accordance with the [MKL Link Line Advisor tool](https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor), which you should use to tailor the arch file to your system and MKL version, selecting  "None" for "usage model of Intel Xeon Phi Coprocessor" option, choosing the relevant Fortran compiler, 32-bit integer interface layer, OpenMP threading, Intel OpenMP runtime (libiomp), the MPI library that matches most closely that on your machine, and selecting ScaLAPACK and BLACS.  

Note the arch file below assumes the ``MKLROOT`` and ``FFTWROOT`` variables are already set in the build environment, as is the case on many systems after loading Intel / MKL and FFTW environment modules. 

```
CC = icc
CPP = 
FC = mpiifort
LD = $(FC)
AR = xiar -r

CPPFLAGS =

DFLAGS = -D__parallel \
         -D__SCALAPACK \
         -D__FFTW3 \
         -D__MKL \
         -D__LIBINT \
         -D__HAS_NO_SHARED_GLIBC 

LIBINTROOT = /path/to/libint/directory

IFLAGS = -I$(FFTWROOT)/include \
         -I$(MKLROOT)/include \  
         -I$(LIBINTROOT)/include

FCFLAGS  = $(DFLAGS) $(IFLAGS) -O2 -xMIC-AVX512 \  
		   -qopenmp -qopenmp-threadprivate=compat \  
		   -pad -qopt-prefetch -funroll-loops -fpp -free

CFLAGS = $(DFLAGS) $(IFLAGS) -O2 

FFTW_LIBS = -L$(FFTWROOT)/lib -lfftw3 -lfftw3_threads

MKL_LIBS = $(MKLROOT)/lib/intel64/libmkl_scalapack_lp64.a \
           -Wl,--start-group \
           $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a \
           $(MKLROOT)/lib/intel64/libmkl_sequential.a \
           $(MKLROOT)/lib/intel64/libmkl_core.a \
           $(MKLROOT)/lib/intel64/libmkl_blacs_intelmpi_lp64.a \
           -Wl,--end-group \
           -lpthread -lm 

LIBINT_LIBS = -L$(LIBINTROOT)/lib -lderiv -lint -lr12

LIBS = $(FFTW_LIBS) \
       $(MKL_LIBS) \  
       $(LIBINT_LIBS) \
       -lstdc++
           
LDFLAGS_C = $(FCFLAGS) $(LIBS) -static-intel -nofor_main 

LDFLAGS = $(FCFLAGS) $(LIBS) -static-intel

```

###### Compiling on Cray machines
On Cray machines the compiler wrappers automatically add the AVX512 instruction flag and take care of linking MKL and FFTW as long as the `PrgEnv-intel` and `fftw` modules are loaded, hence only explicit linking to LIBINT needs to be specified and the arch file can be simpler, as shown below:

```
CC = cc
CPP = 
FC = ftn
LD = $(FC)
AR = xiar -r

CPPFLAGS =

DFLAGS = -D__parallel \
         -D__SCALAPACK \
         -D__FFTW3 \
         -D__MKL \
         -D__LIBINT \
         -D__HAS_NO_SHARED_GLIBC 

LIBINTROOT = /path/to/libint/directory

IFLAGS = -I$(LIBINTROOT)/include

CFLAGS = $(DFLAGS) $(IFLAGS) -O2 

FCFLAGS = $(DFLAGS) $(IFLAGS) -O2 \  
		  -qopenmp -qopenmp-threadprivate=compat \  
		  -pad -qopt-prefetch -funroll-loops -fpp -free

LIBINT_LIBS = -L$(LIBINTROOT)/lib -lderiv -lint -lr12

LIBS = $(LIBINT_LIBS) \  
       -lstdc++
		              
LDFLAGS_C = $(FCFLAGS) $(LIBS) -nofor_main

LDFLAGS = $(FCFLAGS) $(LIBS)

```


### CP2K arch file for CUDA-enabled GPUs
An example arch file for CUDA-enabled GPUs using GNU compilers is shown below. Linking variables for MKL are set in accordance with the [MKL Link Line Advisor tool](https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor), which you should use to tailor the arch file to your system and MKL version,  selecting "None" for "usage model of Intel Xeon Phi Coprocessor" option, choosing the relevant Fortran compiler, 32-bit integer interface layer, OpenMP threading, Intel OpenMP runtime (libiomp), the MPI library that matches most closely that on your machine, and selecting ScaLAPACK and BLACS. 

Note the example arch file below assumes the ``CUDA_PATH``, ``MKLROOT``, and ``FFTWROOT`` variables are already set in the build environment, as is the case on many systems after loading CUDA, Intel / MKL and FFTW environment modules.  

The ``-arch`` flag should be adjusted to choose the right option for the particular Nvidia GPU architecture in question. For example ``-arch sm35`` matches the Tesla K40 and K80 GPU architectures. 

#### Compiling CP2K for GPU on compute nodes
##### Using GCC

```
NVCC = nvcc
CC = 
CPP =
FC = mpif90
LD = $(FC)
AR = ar -r

CPPFLAGS = 

DFLAGS = -D__parallel \  
         -D__SCALAPACK \
         -D__MKL \
         -D__LIBINT \
         -D__HAS_NO_SHARED_GLIBC 

CUDA_FLAGS = -D__ACC -D__DBCSR_ACC -D__PW_CUDA

DFLAGS += $(CUDA_FLAGS)

NVFLAGS = $(DFLAGS) -g -O3 -arch sm_35 

LIBINTROOT = /path/to/libint/directory

IFLAGS = -I$(FFTWROOT)/include \
         -I$(MKLROOT)/include \  
         -I$(LIBINTROOT)/include

CFLAGS = $(DFLAGS) $(IFLAGS) -O3

FCFLAGS = $(DFLAGS) $(IFLAGS) -O3 -march=native \  
          -fopenmp -ffree-form -ffast-math -funroll-loops -fno-tree-vectorize \  
          -fno-omit-frame-pointer -Waliasing -Wampersand -Wc-binding-type \  
          -Wconversion -Wintrinsic-shadow -Wintrinsics-std -Wline-truncation \  
          -Wno-tabs -Wtarget-lifetime -Wrealloc-lhs-all -Wunderflow \  
          -Wunused-but-set-variable -Wunused-variable -ffree-line-length-512 \  
          -std=f2003

MKL_FCFLAGS = -m64

FCFLAGS += $(MKL_FCFLAGS)

FFTW_LIBS = -L$(FFTWROOT)/lib -lfftw3 -lfftw3_threads

CUDA_LIBS = -L$(CUDA_PATH)/lib64 -lcudart -lcublas -lcufft -lrt         

MKL_LIBS = $(MKLROOT)/lib/intel64/libmkl_scalapack_lp64.a \  
           -Wl,--start-group \  
           $(MKLROOT)/lib/intel64/libmkl_gf_lp64.a \  
           $(MKLROOT)/lib/intel64/libmkl_intel_thread.a \
           $(MKLROOT)/lib/intel64/libmkl_core.a \  
           $(MKLROOT)/lib/intel64/libmkl_blacs_intelmpi_lp64.a \  
           -Wl,--end-group \  
           -liomp5 -lpthread -lm -ldl

LIBINT_LIBS = -L$(LIBINTROOT)/lib -lderiv -lint -lr12
           
LIBS = $(FFTW_LIBS) \  
       $(CUDA_LIBS) \  
       $(MKL_LIBS) \  
       $(LIBINT_LIBS) \  
       -lstdc++ 
       
LDFLAGS_C = $(FCFLAGS) $(LIBS) -static

LDFLAGS = $(FCFLAGS) $(LIBS)

```

#### Cross-compiling CP2K for GPU compute nodes
If you are cross-compiling, follow the example arch file shown above for compiling on compute nodes but instead of ``-march=native`` choose an option that matches the processor architecture of your system's compute nodes. 



## Running Test Case A - LiH-HFX benchmark
Test Case A is the LiH-HFX benchmark included in the CP2K distribution and consists of a DFT energy calculation for a 216 atom LiH crystal. 

The following files should be copied from inside the CP2K distribution to a working directory accessible from the compute nodes:

``./tests/QS/benchmark_HFX/LiH/*``

Provided input files include:

- ``input_bulk_HFX_3.inp``: input for the actual benchmark HFX run
- ``input_bulk_B88_3.inp``: needed to generate an initial wave function (wfn file) for the HFX run


### Job scripts

##### B88
Essential steps in the job script for the first, preparatory stage of the benchmark are, using Cray's ``aprun`` parallel application launcher as an example:

```
exe=/path_to_cp2k_distribution/exe/ARCH.VERSION/cp2k.psmp
benchmark=input_bulk_B88_3

export OMP_NUM_THREADS=1
export KMP_AFFINITY=none
aprun -n $TOTAL_MPI_PROCS -N $MPI_PROCS_PER_NODE -d ${OMP_NUM_THREADS} ${exe} -i ${benchmark}.inp -o ${benchmark}.log

cp LiH_bulk_3-RESTART.wfn B88.wfn
```

##### HFX
Again using Cray's ``aprun``, this time with 6 OpenMP threads per MPI rank as an example, key steps in the HFX benchmark run are:

```
exe=/path_to_cp2k_distribution/exe/ARCH.VERSION/cp2k.psmp
benchmark=input_bulk_HFX_3

export OMP_NUM_THREADS=6
export KMP_AFFINITY=none
aprun -n $TOTAL_MPI_PROCS -N $MPI_PROCS_PER_NODE -d ${OMP_NUM_THREADS} ${exe} -i ${benchmark}.inp -o ${benchmark}.log

```
### Timing
The benchmark time may be extracted from the application's runtime profile printed in the log file ``input_bulk_HFX_3.log`` by executing the command

```
grep " CP2K   " input_bulk_HFX_3.log
```
and taking the value shown in the final column, which corresponds to the runtime of the entire application in seconds. 

### Notes
The amount of memory available per MPI process must be altered according to the number of MPI processes being used. If this is not done the benchmark will crash with an out of memory (OOM) error. The input file keyword ``MAX_MEMORY`` in ``input_bulk_HFX_3.inp`` needs to be changed as follows:  

```
MAX_MEMORY 14000
```  

should be changed to  

```
MAX_MEMORY new_value
```  

The new value of ``MAX_MEMORY`` is chosen by dividing the total amount of memory available on a node by the number of MPI processes being used per node. 

If a shorter runtime is desirable, the following line in ``input_bulk_HFX_3.inp``:  

```
MAX_SCF 20
```  

may be changed to  

```
MAX_SCF 1
```  

in order to reduce the maximum number of SCF cycles and hence the execution time. 

If the runtime or required memory needs to be reduced so the benchmark can run on a smaller number of nodes, the OPT1 basis set can be used instead of the default OPT2. To this end, the line  

```
BASIS_SET OPT2
```  

in ``input_bulk_B88_3.inp`` and in ``input_bulk_HFX_3.inp`` should be changed to  
  
```
BASIS_SET OPT1
```  



## Running Test Case B - H2O-DFT-LS benchmark
Test Case B is the H2O-DFT-LS benchmark included in the CP2K distribution and consists of a DFT energy calculation for 2048 water molecules. 

The following files should be copied from inside the CP2K distribution to a working directory accessible from the compute nodes:

```
./tests/QS/benchmark_DM_LS/*
```