Commit 77107564 authored by Cedric Jourdain's avatar Cedric Jourdain 🐵
Browse files

Merge PRACE_UEABS_Specfem3D_summary and README.md

parent 40be11e1
# Specfem 3D globe -- Bench readme
Note that this guide is still a work in progress and is currently being reviewed.
## Summary Version
1.0
## General description
The software package SPECFEM3D simulates three-dimensional global and regional seismic wave propagation based upon the spectral-element method (SEM). All SPECFEM3D_GLOBE software is written in Fortran90 with full portability in mind, and conforms strictly to the Fortran95 standard. It uses no obsolete or obsolescent features of Fortran77. The package uses parallel programming based upon the Message Passing Interface (MPI).
......@@ -17,32 +19,37 @@ In many geological models in the context of seismic wave propagation studies (ex
* Test Case C: https://repository.prace-ri.eu/git/UEABS/ueabs/tree/master/specfem3d/test_cases/SPECFEM3D_TestCaseC
* Run instructions: https://repository.prace-ri.eu/git/UEABS/ueabs/blob/r1.3/specfem3d/SPECFEM3D_Run_README.txt
## Get the source
## Purpose of Benchmark
The software package SPECFEM3D_GLOBE simulates three-dimensional global and regional seismic wave propagation and performs full waveform imaging (FWI) or adjoint tomography based upon the spectral-element method (SEM).
The test cases simulate the earthquake of June 1994 in Northern Bolivia at a global scale with the global shear-wave speed model named s362ani.
Test Case A is designed to run on Tier-1 sized systems (up to around 1,000 x86 cores, or equivalent), Test Case B is designed to run on Tier-0 sized systems (up to around 10,000 x86 cores, or equivalent) and finally the test case C is designed to run on PCP prototypes (up to around 100 cores, or equivalent).
Clone the repository in a location of your choice, let's say $HOME.
## Mechanics of Building Benchmark
### Get the source
Clone the repository in a location of your choice.
Download Specfem3D_Globe software package :
```shell
cd $HOME
git clone https://github.com/geodynamics/specfem3d_globe.git
```
Then use a fixed and stable version of specfem3D_globe (the one of October 31, 2017
for example, see https://github.com/geodynamics/specfem3d_globe/commits/master)
```shell
cd $HOME/specfem3d_globe
cd specfem3d_globe
git checkout b1d6ba966496f269611eff8c2cf1f22bcdac2bd9
```
If this is not done, clone the ueabs repository.
```shell
cd $HOME
git clone https://repository.prace-ri.eu/git/UEABS/ueabs.git
```
In the specfem3D folder of this repo, you will find test cases in the test_cases folder,
you will also find environment and submission scripts templates for several machines
## Load the environment
### Define the environment
You will need a fortran and a C compiler and a MPI library.
The following variables are relevent to compile the code:
**a.** You will need a Fortran and a C compiler and a MPI library and it is recommended that you explicitly specify the appropriate command names for your Fortran compiler in your .bashrc or your .cshrc file (or directly in your submission file). To be exhaustive here are the relevant variables to compile the code:
- `LANG=C`
- `FC`
......@@ -50,9 +57,7 @@ The following variables are relevent to compile the code:
- `CC`
- `MPICC`
Compiling with CUDA to run on GPUs, you will also need to load the cuda environment
and set the two following variables
**b.** To be able to run on GPUs, you must define the CUDA environment by setting the following two variables:
- `CUDA_LIB`
- `CUDA_INC`
......@@ -71,28 +76,29 @@ export MPICC=`which mpicc`
export CUDA_LIB="$CUDAROOT/lib64"
export CUDA_INC="$CUDAROOT/include"
```
Once again, you will find in the specfem3D folder of this repo a folder named env,
You will find in the specfem3D folder of this repo a folder named env,
with files named env_x which gives examples of the environment used on several supercomputers
during the last benchmark campaign
## Compile specfem
As arrays are staticaly declared, you will need to compile specfem once for each
test case with the right `Par_file`
On some environement, depending on MPI configuration you will need to replace
`use mpi` statement with `include mpif.h`, use the script and prodedure commented
below.
**c.** To define the optimization specific to the target architecture, you will need the environment variables FCFLAGS and CFLAGS.
First you will have to configure.
### Configuration step
To configure specfem3D_Globe use the configure script, this script assumes that
you will compile the code on the same kind of hardware as the machine on which
you will run it. **As arrays are staticaly declared, you will need to compile specfem
once for each test case with the right `Par_file`** which is the parameter file of specfem3D.
**On GPU platform** you will have to add the following arguments to the
configure:`--build=ppc64 --with-cuda=cuda5`.
To use the **shared memory parallel programming** model of specfem3D we will
specify `--enable-openmp` configure option.
```shell
cp -r $HOME/specfem3d_globe specfem_compil_${test_case_id}
cp $HOME/bench_spec/test_case_${test_case_id}/DATA/Par_file specfem_compil_${test_case_id}/DATA/
**On GPU platform** you will need to add the following arguments to the configure
`--build=ppc64 --with-cuda=cuda5` and you will need to set the `GPU_MODE = .true.`
in the parameter file `Par_file`
cd specfem_compil_${test_case_id}
On some environement, depending on MPI configuration you will need to replace
`use mpi` statement with `include mpif.h`, use the script and prodedure commented below.
```shell
### replace `use mpi` if needed ###
# cd utils
# perl replace_use_mpi_with_include_mpif_dot_h.pl
......@@ -113,6 +119,7 @@ CPPFLAGS = -I${SETUP} -DFORCE_VECTORIZATION -xMIC-AVX512
Note: Be careful, in most machines login node does not have the same instruction set so, in order to compile with the right instruction set, you'll have to compile on a compute node (salloc + ssh)
### Compilation
Finally compile with make:
```shell
make clean
......@@ -121,7 +128,24 @@ make all
**-> You will find in the specfem folder of ueabs repository the file "compile.sh" which is an compilation script template for several machines (different architectures : KNL, SKL, Haswell and GPU)**
## Launch specfem
## Mechanics of running
Input for the mesher (and the solver) is provided through the parameter file Par_file, which resides in the subdirectory DATA. Before running the mesher, a number of parameters need to be set in the Par_file. The solver calculates seismograms for 129 stations, and simulations are run for a record length of 3 minutes 30 for test case A, 10 minutes for test case B and one minute for test case C.
The different test cases correspond to different meshes of the earth. The size of the mesh is determined by a combination of following variables: NCHUNKS, the number of chunks in the cubed sphere (6 for global simulations), NPROC_XI, the number of processors or slices along one chunk of the cubed sphere and NEX_XI, the number of spectral elements along one side of a chunk in the cubed sphere. These three variables give us the number of degrees of freedom of the mesh and determine the amount of memory needed per core. The Specfem3D solver must be recompiled each time we change the mesh size because the solver uses a static loop size and the compilers know the size of all loops only at the time of compilation and can therefore optimize them efficiently.
- Test case A runs with `96 MPI` tasks using hybrid parallelization (MPI+OpenMP or MPI+OpenMP+Cuda depending on the system tested) and has the following mesh characteristics: NCHUNKS=6, NPROC_XI=4 and NEX_XI=384.
- Test Case B runs with `1536 MPI` tasks using hybrid parallelization and has the following mesh characteristics: NCHUNKS=6, NPROC_XI=16 and NEX_XI=384.
- Test Case C runs with `6 MPI` tasks using hybrid parallelization and has the following mesh characteristics: NCHUNKS=6, NPROC_XI=1 and NEX_XI=64.
Once the parameter file is correctly defined, to run the test cases, copy the `Par_file`,
`STATIONS` and `CMTSOLUTION` files defining one of the three test cases (A, B or
C cf https://repository.prace-ri.eu/git/UEABS/ueabs/-/tree/r2.2-dev/specfem3d/test_cases)
into the SPECFEM3D_GLOBE/DATA directory. Then use the xmeshfem3D binary (located
in the bin directory) to mesh the domain and xspecfem3D to solve the problem using
the appropriate command to run parallel jobs (srun, ccc_mprun, mpirun…).
```shell
srun bin/xmeshfem3D
srun bin/xspecfem3D
```
You can use or be inspired by the submission script template in the job_script folder using the appropriate job submission command :
- qsub for pbs job,
......@@ -130,10 +154,10 @@ You can use or be inspired by the submission script template in the job_script f
- llsubmit for LoadLeveler job.
## Gather results
The relevant metric for this benchmark is time for the solver. Using slurm, it is
easy to gather as each `mpirun` or `srun` is interpreted as a step wich is already
timed. So the command line `sacct -j <job_id>` allows you to catch the metric.
Or you can find more precise timing info at the end of this output file : specfem3d_globe/OUTPUT_FILES/output_solver.txt
\ No newline at end of file
## Verification of results
The relevant metric for this benchmark is time for the solver. Using slurm, it is
easy to gather as each `mpirun` or `srun` is interpreted as a step which is already
timed. So the command line `sacct -j <job_id>` allows you to catch the metric.
The output of the mesher (“output_mesher.txt”) and of the solver (“output_solver.txt”)
can be find in the OUTPUT_FILES directory. These files contains physical values
and timing values that are more accurate than those collected by slurm.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment