========================================================================
README file for PRACE Accelerator Benchmark Code PFARM (stage EXDIG, program RMX95)
========================================================================
Author: Andrew Sunderland (andrew.sunderland@stfc.ac.uk).

The code download should contain the following directories:

benchmark/RMX_MPI_OMP: RMX source files for running on Host or KNL (using serial or threaded LAPACK or MKL)
benchmark/RMX_MAGMA_GPU: RMX source for running on CPU/GPU nodes using MAGMA
benchmark/run: Run directory with input files
benchmark/xdr: XDR library src files and static XDR library file
benchmark/data: Data files for the benchmark test cases

The code uses the eXternal Data Representation library (XDR) for cross-platform
compatibility of unformatted data files. The XDR source files are provided with this code bundle.
and can be obtained from various sources, including
http://meteora.ucsd.edu/~pierce/fxdr_home_page.html
http://people.redhat.com/rjones/portablexdr/

----------------------------------------------------------------------------
* Installing (MAGMA GPU Only)
Download MAGMA (current version magma-2.2.0)  from http://icl.utk.edu/magma/
Install MAGMA : Modify the make.inc file to indicate your C/C++
 compiler, Fortran compiler, and determine where CUDA, CPU BLAS, and 
 LAPACK are installed on your system. Refer to MAGMA documentation for further details
----------------------------------------------------------------------------
* Install XDR
Build XDR library: 
update DEFS file for your compiler and environment
$> cd xdr
$> make
(ignore warnings related to float/double type mismatches in xdr_rmat64.c - this is not relevant for this benchmark) 
The validity of the XDR library can be tested by running test_xdr
$> ./test_xdr

ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
* Install RMX_MPI_OMP
$> cd RMX_MPI_OMP
Update DEFS file for your setup, ensuring you are linking to a LAPACK or MKL library (or equivalent).
This is usually facilitated by e.g. compiling with -mkl=parallel (Intel compiler) or loading the appropriate library modules.
$> make

* Install RMX_MAGMA_GPU
Set MAGMADIR, CUDADIR environment variables to point to MAGMA and CUDA installations
$> cd RMX_MAGMA_GPU
Update DEFS file for your setup
$> make

----------------------------------------------------------------------------
* Run RMX
==========
The RMX application can be run by running the executable "rmx95"
For the FEIII dataset, the program requires the following input files from the data directory linked to the run directory :
phzin.ctl
XJTARMOM
HXJ030

These files are located in benchmark/run
A guide to each of the variables in the namelist in phzin.ctl can be found at:
https://hpcforge.org/plugins/mediawiki/wiki/pfarm/images/9/99/Phz_rep.pdf
However, it is recommended that these inputs are not changed for the benchmark runs and
problem size, runtime etc, are better controlled via the environment variables listed below.

* Run on CPUs (or KNLs)
=======================
A typical PBS script to run the RMX_MPI_OMP benchmark on 4 KNL nodes (4 MPI tasks with 64 threads per MPI task) is listed below:
Settings will vary according to your local environment.

#PBS -N rmx95_4x64
#PBS -l select=4
#PBS -l walltime=01:00:00
#PBS -A my_account_id

cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=64

# Set some code-specific environment variables (for details see below) e.g.:
export RMX_NSECT_FINE=4
export RMX_NSECT_COARSE=4
export RMX_NL_FINE=12
export RMX_NL_COARSE=6

# Run on 4 nodes with 1 MPI task per node and 24 OpenMP tasks per node
aprun -N 1 -n 4 -d $OMP_NUM_THREADS ./rmx95

* Run on CPU/GPU nodes
======================
A typical PBS script to run the RMX_MPI_GPU benchmark on 4 CPU nodes (with 2 GPUs per CPU) is listed below:
Settings will vary according to your local environment.

#PBS -N rmx95_4MPIx2GPU
#PBS -l select=4
#PBS -l walltime=01:00:00
#PBS -A my_account_id

cd $PBS_O_WORKDIR
# Set number of GPUs per node to use (MAGMA auto-parallelises over these)
export RMX_NGPU=2  

# Set some code-specific environment variables (for details see below) e.g.:
export RMX_NSECT_FINE=4
export RMX_NSECT_COARSE=4
export RMX_NL_FINE=12
export RMX_NL_COARSE=6

# Run on 4 nodes with 1 MPI task per node and 2 GPUs per node
aprun -N 1 -n 4 ./rmx95

----------------------------------------------------------------------------

* Run-time environment variable settings

The following environmental variables that e.g. can be set inside the script allow the H sector matrix 
to easily change dimensions and also allows the number of sectors to change when undertaking benchmarks.
These can be adapted by the user to suit benchmark load requirements e.g. short vs long runs.
Each MPI Task will pickup a sector calculation which will then be distributed amongst available threads per node (for CPU and KNL) or offloaded (for GPU).
The distribution among MPI tasks is simple round-robin.
 
RMX_NGPU : refers to the number of shared GPUs per node (only for RMX_MAGMA_GPU)
RMX_NSECT_FINE : sets the number of sectors for the Fine region (it is recommended to set this to a low number if the sector Hamiltonian matrix dimension is large).
RMX_NSECT_COARSE : sets the number of sectors for the Coarse region (it is recommended to set this to a low number if the sector Hamiltonian matrix dimension is large).
RMX_NL_FINE : sets the number of basis functions for the Fine region sector calculations (this will determine the size of the sector Hamiltonian matrix). 
RMX_NL_COARSE : sets the number of basis functions for the Coarse region sector calculations (this will determine the size of the sector Hamiltonian matrix). 
Hint: To aid scaling across nodes, the number of MPI tasks in the job script should ideally be a factor of RMX_NSECT_FINE.

For representative test cases: 
RMX_NL_FINE should take values in the range 6:25
RMX_NL_COARSE should take values in the range 5:10 

For accuracy reasons, RMX_NL_FINE should always be great than RMX_NL_COARSE. 
The following value pairs for RMX_NL_FINE and RMX_NL_COARSE provide representative calculations:

12,6
14,8
16,10
18,10
20,10
25,10

If RMX_NSECT and RMX_NL variables are not set, the benchmark code defaults to calculating NL and NSECT, giving:
RMX_NSECT_FINE=5
RMX_NSECT_COARSE=20
RMX_NL_FINE=12
RMX_NL_COARSE=6

* Results

1 AMPF file will be created for each fine-region sector
1 AMPC file will be created for each coarse-region sector

All output AMPF files will be the same size and all output AMPC files will be the same size (bytes).

The Hamiltonian matrix dimension will be output along 
with the Wallclock time it takes to do each individual DSYEVD call.

Performance is measured in Wallclock time and is displayed 
on the screen or output log at the end of the run. 


----------------------------------------------------------------------------