Update pfarm/PFARM_Build_Run_README.txt

pfarm/PFARM_Build_Run_README.txt

+========================================================================
+README file for PRACE Accelerator Benchmark Code PFARM (stage EXDIG, program RMX95)
+========================================================================
+Author: Andrew Sunderland (a.g.sunderland@stfc.ac.uk).
+
+The code download should contain the following directories:
+
+benchmark/RMX_MPI_OMP: RMX source files for running on Host or KNL (using serial or threaded LAPACK or MKL)
+benchmark/RMX_MAGMA_GPU: RMX source for running on CPU/GPU nodes using MAGMA
+benchmark/run: Run directory with input files
+benchmark/xdr: XDR library src files and static XDR library file
+benchmark/data: Data files for the benchmark test cases
+
+The code uses the eXternal Data Representation library (XDR) for cross-platform
+compatibility of unformatted data files. The XDR source files are provided with this code bundle.
+and can be obtained from various sources, including
+http://meteora.ucsd.edu/~pierce/fxdr_home_page.html
+http://people.redhat.com/rjones/portablexdr/
+
+----------------------------------------------------------------------------
+* Installing (MAGMA GPU Only)
+Download MAGMA (current version magma-2.2.0)  from http://icl.utk.edu/magma/
+Install MAGMA : Modify the make.inc file to indicate your C/C++
+ compiler, Fortran compiler, and determine where CUDA, CPU BLAS, and 
+ LAPACK are installed on your system. Refer to MAGMA documentation for further details
+----------------------------------------------------------------------------
+* Install XDR
+Build XDR library: 
+update DEFS file for your compiler and environment
+$> cd xdr
+$> make
+(ignore warnings related to float/double type mismatches in xdr_rmat64.c - this is not relevant for this benchmark) 
+The validity of the XDR library can be tested by running test_xdr
+$> ./test_xdr
+
+——————————————————————————————————————————————————————————————————————————
+* Install RMX_MPI_OMP
+$> cd RMX_MPI_OMP
+Update DEFS file for your setup, ensuring you are linking to a LAPACK or MKL library (or equivalent).
+This is usually facilitated by e.g. compiling with -mkl=parallel (Intel compiler) or loading the appropriate library modules.
+$> make
+
+* Install RMX_MAGMA_GPU
+Set MAGMADIR, CUDADIR environment variables to point to MAGMA and CUDA installations
+$> cd RMX_MAGMA_GPU
+Update DEFS file for your setup
+$> make
+
+----------------------------------------------------------------------------
+* Run RMX
+==========
+The RMX application can be run by running the executable "rmx95"
+For the FEIII dataset, the program requires the following input files from the data directory linked to the run directory :
+phzin.ctl
+XJTARMOM
+HXJ030
+
+These files are located in benchmark/run
+A guide to each of the variables in the namelist in phzin.ctl can be found at:
+https://hpcforge.org/plugins/mediawiki/wiki/pfarm/images/9/99/Phz_rep.pdf
+However, it is recommended that these inputs are not changed for the benchmark runs and
+problem size, runtime etc, are better controlled via the environment variables listed below.
+
+* Run on CPUs (or KNLs)
+=======================
+A typical PBS script to run the RMX_MPI_OMP benchmark on 4 KNL nodes (4 MPI tasks with 64 threads per MPI task) is listed below:
+Settings will vary according to your local environment.
+
+#PBS -N rmx95_4x64
+#PBS -l select=4
+#PBS -l walltime=01:00:00
+#PBS -A my_account_id
+
+cd $PBS_O_WORKDIR
+export OMP_NUM_THREADS=64
+
+# Set some code-specific environment variables (for details see below) e.g.:
+export RMX_NSECT_FINE=4
+export RMX_NSECT_COARSE=4
+export RMX_NL_FINE=12
+export RMX_NL_COARSE=6
+
+# Run on 4 nodes with 1 MPI task per node and 24 OpenMP tasks per node
+aprun -N 1 -n 4 -d $OMP_NUM_THREADS ./rmx95
+
+* Run on CPU/GPU nodes
+======================
+A typical PBS script to run the RMX_MPI_GPU benchmark on 4 CPU nodes (with 2 GPUs per CPU) is listed below:
+Settings will vary according to your local environment.
+
+#PBS -N rmx95_4MPIx2GPU
+#PBS -l select=4
+#PBS -l walltime=01:00:00
+#PBS -A my_account_id
+
+cd $PBS_O_WORKDIR
+# Set number of GPUs per node to use (MAGMA auto-parallelises over these)
+export RMX_NGPU=2  
+
+# Set some code-specific environment variables (for details see below) e.g.:
+export RMX_NSECT_FINE=4
+export RMX_NSECT_COARSE=4
+export RMX_NL_FINE=12
+export RMX_NL_COARSE=6
+
+# Run on 4 nodes with 1 MPI task per node and 2 GPUs per node
+aprun -N 1 -n 4 ./rmx95
+
+----------------------------------------------------------------------------
+
+* Run-time environment variable settings
+
+The following environmental variables that e.g. can be set inside the script allow the H sector matrix 
+to easily change dimensions and also allows the number of sectors to change when undertaking benchmarks.
+These can be adapted by the user to suit benchmark load requirements e.g. short vs long runs.
+Each MPI Task will pickup a sector calculation which will then be distributed amongst available threads per node (for CPU and KNL) or offloaded (for GPU).
+The distribution among MPI tasks is simple round-robin.
+ 
+RMX_NGPU : refers to the number of shared GPUs per node (only for RMX_MAGMA_GPU)
+RMX_NSECT_FINE : sets the number of sectors for the Fine region (it is recommended to set this to a low number if the sector Hamiltonian matrix dimension is large).
+RMX_NSECT_COARSE : sets the number of sectors for the Coarse region (it is recommended to set this to a low number if the sector Hamiltonian matrix dimension is large).
+RMX_NL_FINE : sets the number of basis functions for the Fine region sector calculations (this will determine the size of the sector Hamiltonian matrix). 
+RMX_NL_COARSE : sets the number of basis functions for the Coarse region sector calculations (this will determine the size of the sector Hamiltonian matrix). 
+Hint: To aid scaling across nodes, the number of MPI tasks in the job script should ideally be a factor of RMX_NSECT_FINE.
+
+For representative test cases: 
+RMX_NL_FINE should take values in the range 6:25
+RMX_NL_COARSE should take values in the range 5:10 
+
+For accuracy reasons, RMX_NL_FINE should always be great than RMX_NL_COARSE. 
+The following value pairs for RMX_NL_FINE and RMX_NL_COARSE provide representative calculations:
+
+12,6
+14,8
+16,10
+18,10
+20,10
+25,10
+
+If RMX_NSECT and RMX_NL variables are not set, the benchmark code defaults to calculating NL and NSECT, giving:
+RMX_NSECT_FINE=5
+RMX_NSECT_COARSE=20
+RMX_NL_FINE=12
+RMX_NL_COARSE=6
+
+* Results
+
+1 AMPF file will be created for each fine-region sector
+1 AMPC file will be created for each coarse-region sector
+
+All output AMPF files will be the same size and all output AMPC files will be the same size (bytes).
+
+The Hamiltonian matrix dimension will be output along 
+with the Wallclock time it takes to do each individual DSYEVD call.
+
+Performance is measured in Wallclock time and is displayed 
+on the screen or output log at the end of the run. 
+
+
+----------------------------------------------------------------------------
\ No newline at end of file