Commit 70d4a138 authored by Cedric Jourdain's avatar Cedric Jourdain 🐵
Browse files

Update README.md

parent 7639b04d
# Specfem 3D globe -- Bench readme # Specfem 3D globe -- Bench readme
Note that this guide is still a work in progress and is currently being reviewed.
## Get the source ## Get the source
Clone the repository in a location of your choice, let's say $HOME. Clone the repository in a location of your choice, let's say $HOME.
...@@ -62,7 +60,8 @@ during the last benchmark campaign ...@@ -62,7 +60,8 @@ during the last benchmark campaign
## Compile specfem ## Compile specfem
As arrays are staticaly declared, you will need to compile specfem once for each As arrays are staticaly declared, you will need to compile specfem once for each
test case with the right `Par_file` test case with the right `Par_file`. Indeed, input for the mesher (and the solver) is provided through the parameter file Par_file, which resides in the
subdirectory DATA of specfem3D\_Globe. Before running the mesher, a number of parameters need to be set in the Par\_file. In our case, the Par\_file for each test case is provided in the subdirectories test_cases.
On some environement, depending on MPI configuration you will need to replace On some environement, depending on MPI configuration you will need to replace
`use mpi` statement with `include mpif.h`, use the script and prodedure commented `use mpi` statement with `include mpif.h`, use the script and prodedure commented
below. below.
...@@ -73,10 +72,9 @@ First you will have to configure. ...@@ -73,10 +72,9 @@ First you will have to configure.
configure:`--build=ppc64 --with-cuda=cuda5`. configure:`--build=ppc64 --with-cuda=cuda5`.
```shell ```shell
cp -r $HOME/specfem3d_globe specfem_compil_${test_case_id} cp -r $HOME/specfem3d_globe specfem_build_${test_case_id}
cp $HOME/bench_spec/test_case_${test_case_id}/DATA/Par_file specfem_compil_${test_case_id}/DATA/ cp $HOME/ueabs/specfem3d/test_cases/SPECFEM3D_${test_case_id}/Par_file specfem_build_${test_case_id}/DATA/
cd specfem_build_${test_case_id}
cd specfem_compil_${test_case_id}
### replace `use mpi` if needed ### ### replace `use mpi` if needed ###
# cd utils # cd utils
...@@ -84,27 +82,35 @@ cd specfem_compil_${test_case_id} ...@@ -84,27 +82,35 @@ cd specfem_compil_${test_case_id}
# cd .. # cd ..
#################################### ####################################
./configure --prefix=$PWD # GPU platform
``` ./configure --prefix=$PWD --build=ppc64 --with-cuda=cuda5
# Otherwise
#./configure --prefix=$PWD --enable-openmp
**On Xeon Phi**, since support is recent you should replace the following variables ```
values in the generated Makefile:
```Makefile Depending on the architecture you will have to export (before configuring) different options for the environment variables related to the compilation (CFLAGS, CPPFLAGS, FCFLAGS...) (or you can modify the values of the variables in the generated Makefile).
FCFLAGS = -g -O3 -qopenmp -xMIC-AVX512 -DUSE_FP32 -DOPT_STREAMS -align array64byte -fp-model fast=2 -traceback -mcmodel=large Here is an example of the variables to (re)define **On Xeon Phi**
```Makefile/environnment
FCFLAGS=" -g -O3 -qopenmp -xMIC-AVX512 -DUSE_FP32 -DOPT_STREAMS -fp-model fast=2 -traceback -mcmodel=large -fma -align array64byte -finline-functions -ipo"
CFLAGS=" -g -O3 -xMIC-AVX512 -fma -align -finline-functions -ipo"
FCFLAGS_f90 = -mod ./obj -I./obj -I. -I. -I${SETUP} -xMIC-AVX512 FCFLAGS_f90 = -mod ./obj -I./obj -I. -I. -I${SETUP} -xMIC-AVX512
CPPFLAGS = -I${SETUP} -DFORCE_VECTORIZATION -xMIC-AVX512 CPPFLAGS = -I${SETUP} -DFORCE_VECTORIZATION -xMIC-AVX512
``` ```
Another example for **Skylake** architecture:
```Makefile/environnment
export FCFLAGS=" -g -O3 -qopenmp -xCORE-AVX512 -mtune=skylake -ipo -DUSE_FP32 -DOPT_STREAMS -fp-model fast=2 -traceback -mcmodel=large"
export CFLAGS=" -g -O3 -xCORE-AVX512 -mtune=skylake -ipo"
```
Note: Be careful, in most machines login node does not have the same instruction set so, in order to compile with the right instruction set, you'll have to compile on a compute node (salloc + ssh) Note: Be careful, in most machines login node does not have the same instruction set so, in order to compile with the right instruction set, you'll have to compile on a compute node (salloc + ssh)
Finally compile with make: Finally compile with make:
```shell ```shell
make clean make clean
make all make all
``` ```
**-> You will find in the specfem folder of ueabs repository the file "compile.sh" which is an compilation script template for several machines (different architectures : KNL, SKL, Haswell and GPU)** **-> You will find in the specfem folder of ueabs repository the file "compile.sh" which is an compilation script template for several machines (different architectures : KNL, SKL, Power 8, Haswell and GPU)**
## Launch specfem ## Launch specfem
...@@ -117,16 +123,5 @@ You can use or be inspired by the submission script template in the job_script f ...@@ -117,16 +123,5 @@ You can use or be inspired by the submission script template in the job_script f
## Gather results ## Gather results
The relevant metric for this benchmark is time for the solver. Using slurm, it is The relevant metric for this benchmark is time for the solver, you can find it at the end of this output file : specfem3d_globe/OUTPUT_FILES/output_solver.txt.
easy to gather as each `mpirun` or `srun` is interpreted as a step wich is already Using slurm, it is easy to gather as each `mpirun` or `srun` is interpreted as a step wich is already timed. So the command line `sacct -j <job_id>` allows you to catch the metric.
timed. So the command line `sacct -j <job_id>` allows you to catch the metric.
Or you can find more precise timing info at the end of this output file : specfem3d_globe/OUTPUT_FILES/output_solver.txt
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment