# Code_Saturn

## GPU Version

Code_Saturne 4.2.2 is linked to PETSC developer's version, in order to benefit from
its GPU implementation. Note that the normal release of PETSC does not support GPU.

### Installation

The version has been tested for K80s, and with the following settings:

 * OPENMPI 2.0.2
 * GCC 4.8.5
 * CUDA 7.5

To install Code_Saturne 4.2.2, 4 libraries are required, BLAS, LAPACK, SOWING and CUSP.
The tests have been carried out with lapack-3.6.1 for BLAS and LAPACK, sowing-1.1.23-p1
for SOWING and cusplibrary-0.5.1 for CUSP.

PETSC is first installed, and `PATH_TO_PETSC`, `PATH_TO_CUSP`, `PATH_TO_SOWING`, `PATH_TO_LAPACK`
have to be updated in `INSTALL_PETSC_GPU_sm37` under `petsc-petsc-a31f61e8abd0`
PETSC is configured for K80s, `./INSTALL_PETSC_GPU_sm37` is used from `petsc-petsc-a31f61e8abd0`
It is finally compiled and installed, by typing `make` and `make install`.

Before installing Code\_Saturne, adapt `PATH_TO_PETSC` in `InstallHPC.sh` under `SATURNE_4.2.2`
and type `./InstallHPC.sh`

The code should be installed and code_saturne be found under:

```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne
```

And should return:

```
Usage: ./code_saturne <topic>

Topics:
  help
  autovnv
  bdiff
  bdump
  compile
  config
  create
  gui
  info
  run
  salome
  submit

Options:
  -h, --help  show this help message and exit
```

### Test case - Cavity 13M

In `CAVITY_13M.tar.gz` are found the mesh+partitions and 2 sets of subroutines, one for CPU and the
second one for GPU, i.e.:

```
CAVITY_13M/PETSC_CPU/SRC/*
CAVITY_13M/PETSC_GPU/SRC/*
CAVITY_13M/MESH/mesh_input_13M
CAVITY_13M/MESH/partition_METIS_5.1.0/*
```

To prepare a run, it is required to set up a "study" with 2 directories, one for CPU and the other one for GPU
as, for instance:

```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_CAVITY_13M PETSC_CPU
cd NEW_CAVITY_13M
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --case PETSC_GPU
```

The mesh has to be copied from `CAVITY_13M/MESH/mesh_input_13M` into `NEW_CAVITY_13M/MESH/`.
And the same has to be done for `partition_METIS_5.1.0`.

The subroutines contained in `CAVITY_13M/PETSC_CPU/SRC` should be copied into `NEW_CAVITY_13M/PETSC_CPU/SRC` and
the subroutines contained in `CAVITY_13M/PETSC_GPU/SRC` should be copied into `NEW_CAVITY_13M/PETSC_GPU/SRC`.

In each DATA subdirectory of `NEW_CAVITY_13M/PETSC_CPU` and `NEW_CAVITY_13M/PETSC_GPU`, the path
to the mesh+partition has to be set as:

```
cd DATA
cp REFERENCE/cs_user_scripts.py .
```

edit cs_user_scripts.py:

 * At line 138, change `None` to `../MESH/mesh_input_13M`
 * At line 139, change `None` to `../MESH/partition_METIS_5.1.0`

At this stage, everything is set to run both simulations, one for the CPU and the other one for the GPU.

```
cd NEW_CAVITY_13M/PETSC_CPU
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```

Then submit the job.


```
cd NEW_CAVITY_13M/PETSC_GPU
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```

submit the job

## KNL Version

Code_Saturne 4.2.2 is installed for KNLs. It is also linked to PETSC, but the
default linear solvers are the native ones.

### Installation

The installation script is:

```
SATURNE_4.2.2/InstallHPC_with_PETSc.sh
```

The path to PETSC (official released version, and therefore assumed to be installed
on the machine) should be added to the aforementioned script.

After typing `./InstallHPC_with_PETSc.sh` the code should be installed and code_saturne be found under:

```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne
```

And should return:

```
Usage: ./code_saturne <topic>

Topics:
  help
  autovnv
  bdiff
  bdump
  compile
  config
  create
  gui
  info
  run
  salome
  submit

Options:
  -h, --help  show this help message and exit
```

### Running the code

Two cases are dealt with, `TGV_256_CS_OPENMP.tar.gz` to test the native solvers, and
`CAVITY_13M_FOR_KNLs_WITH_PETSC.tar.gz` to test Code_Saturne and PETSC on KNLs

#### First test case: TGV_256_CS_OPENMP

In `TGV_256_CS_OPENMP.tar.gz` are found the mesh and the set of subroutines for Code_Saturne on KNLs, i.e.:

```
TGV_256_CS_OPENMP/MESH/mesh_input_256by256by256
TGV_256_CS_OPENMP/ARCHER_KNL/SRC/*
```

To prepare a run, it is required to set up a "study" as, for instance:

```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_TGV_256_CS_OPENMP KNL
```

The mesh has to be copied from `TGV_256_CS_OPENMP/MESH/mesh_input_256by256by256` into `NEW_TGV_256_CS_OPENMP/MESH/`.

The subroutines contained in `TGV_256_CS_OPENMP/KNL/SRC` should be copied into `NEW_TGV_256_CS_OPENMP/KNL/SRC`.

In the `DATA` subdirectory of `NEW_TGV_256_CS_OPENMP/KNL` the path to the mesh has to be set as:

```
cd DATA
cp REFERENCE/cs_user_scripts.py .
```

Then edit `cs_user_scripts.py`:
At line 138, change `None` to `../MESH/mesh_input_256by256by256`

At this stage, everything is set to run the simulation:

```
cd TGV_256_CS_OPENMP/KNL
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```

And then submit the job.

#### Second test case: CAVITY_13M_FOR_KNLs_WITH_PETSC

In `CAVITY_13M.tar.gz` are found the mesh and the set of subroutines for PETSC and KNLs, i.e.:

```
CAVITY_13M/MESH/mesh_input
CAVITY_13M/KNL/SRC/*
```

To prepare a run, it is required to set up a "study" as, for instance:

```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_CAVITY_13M PETSC_KNL
```

The mesh has to be copied from `CAVITY_13M/MESH/mesh_input` into `NEW_CAVITY_13M/MESH/`.

The subroutines contained in `CAVITY_13M/KNL/SRC` should be copied into `NEW_CAVITY_13M/PETSC_KNL/SRC`.

In the `DATA` subdirectory of `NEW_CAVITY_13M/PETSC_KNL` the path to the mesh has to be set as:

```
cd DATA
cp REFERENCE/cs_user_scripts.py .
```

Then edit `cs_user_scripts.py`
At line 138, change `None` to `../MESH/mesh_input`

At this stage, everything is set to run the simulation:

```
cd NEW_CAVITY_13M/PETSC_KNL
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```

Finaly submit the job