************************************************************************************
Code_Saturne 4.2.2 is linked to PETSC developer's version, in order to benefit from
its GPU implementation. Note that the normal release of PETSC does not support GPU.
************************************************************************************
Installation
************************************************************************************

The version has been tested for K80s, and with the following settings:-

-OPENMPI 2.0.2
-GCC 4.8.5
-CUDA 7.5

To install Code_Saturne 4.2.2, 4 libraries are required, BLAS, LAPACK, SOWING and CUSP.
The tests have been carried out with lapack-3.6.1 for BLAS and LAPACK, sowing-1.1.23-p1
for SOWING and cusplibrary-0.5.1 for CUSP.

PETSC is first installed, and PATH_TO_PETSC, PATH_TO_CUSP, PATH_TO_SOWING, PATH_TO_LAPACK
have to be updated in INSTALL_PETSC_GPU_sm37 under petsc-petsc-a31f61e8abd0
PETSC is configured for K80s, ./INSTALL_PETSC_GPU_sm37 is used from petsc-petsc-a31f61e8abd0
It is finally compiled and installed, by typing make and make install.

Before installing Code_Saturne, adapt PATH_TO_PETSC in InstallHPC.sh under SATURNE_4.2.2
and type ./InstallHPC.sh

The code should be installed and code_saturne be found under:

PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne, which should return:

Usage: ./code_saturne <topic>

Topics:
  help
  autovnv
  bdiff
  bdump
  compile
  config
  create
  gui
  info
  run
  salome
  submit

Options:
  -h, --help  show this help message and exit

************************************************************************************
Test case - Cavity 13M
************************************************************************************

In CAVITY_13M.tar.gz are found the mesh+partitions and 2 sets of subroutines, one for CPU and the
second one for GPU, i.e.:

CAVITY_13M/PETSC_CPU/SRC/*
CAVITY_13M/PETSC_GPU/SRC/*
CAVITY_13M/MESH/mesh_input_13M
CAVITY_13M/MESH/partition_METIS_5.1.0/*

To prepare a run, it is required to set up a "study" with 2 directories, one for CPU and the other one for GPU
as, for instance:

PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_CAVITY_13M PETSC_CPU
cd NEW_CAVITY_13M
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --case PETSC_GPU

The mesh has to be copied from CAVITY_13M/MESH/mesh_input_13M into NEW_CAVITY_13M/MESH/.
And the same has to be done for partition_METIS_5.1.0.

The subroutines contained in CAVITY_13M/PETSC_CPU/SRC should be copied into NEW_CAVITY_13M/PETSC_CPU/SRC and
the subroutines contained in CAVITY_13M/PETSC_GPU/SRC should be copied into NEW_CAVITY_13M/PETSC_GPU/SRC.

In each DATA subdirectory of NEW_CAVITY_13M/PETSC_CPU and NEW_CAVITY_13M/PETSC_GPU, the path
to the mesh+partition has to be set as:

cd DATA
cp REFERENCE/cs_user_scripts.py .
edit cs_user_scripts.py
At line 138, change None to "../MESH/mesh_input_13M"
At line 139, change None to "../MESH/partition_METIS_5.1.0"

At this stage, everything is set to run both simulations, one for the CPU and the other one for the GPU.

cd NEW_CAVITY_13M/PETSC_CPU
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
submit the job

cd NEW_CAVITY_13M/PETSC_GPU
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
submit the job