Provide the Astrophysical community information on the performance and scalability (weak and strong scaling) of the Gadget-4 code associated to three test cases in PRACE Tier-0 supercomputers (JUWELS, MareNostrum4, IRENE-SKL, and IRENE-KNL) and the ARM64 Mont-Blanc 3 prototype (Dibona).
GADGET-4 supports collisionless simulations and smoothed particle hydrodynamics on massively parallel computers. All communication between concurrent execution processes is done either explicitly by means of the message passing interface (MPI), or implicitly through shared-memory accesses on processes on multi-core nodes. The code is mostly written in ISO C++ (assuming the C++11 standard), and should run on all parallel platforms that support at least MPI-3.
Provide the Astrophysical community information on the performance and scalability (weak and strong scaling) of the Gadget-4 code associated to three test cases in PRACE Tier-0 supercomputers (JUWELS, MareNostrum4, IRENE-SKL, and IRENE-KNL).
## Characteristics of Benchmark
GADGET-4 was compiled with the optimisation level O3. In order to have a proper scalability analysis the tests we carried out with one MPI-task per core and 16 tasks per CPU. Hence, a total of 32 tasks (cores) dedicated to the calculations were used per compute node. An extra core to handle the MPI communications was used per compute node. The tests were carried out using two modes: Intel and GCC compiled MPI API and libraries (HPF5, GSL, FFTW3).
GADGET-4 was compiled with C++ with the optimisation level O3, MPI (e.g., OpenMPI, Intel MPI) and the libraries HDF5, GSL, and FFTW3. The tests were carried out using two modes: Intel and GCC compiled MPI API and libraries. In order to have a proper scalability analysis the tests we carried out with one MPI-task per core and 16 tasks per CPU. Hence, a total of 32 tasks (cores) dedicated to the calculations were used per compute node. An extra core to handle the MPI communications was used per compute node.
## Mechanics of Building Benchmark
Building the GADGET code requires a compiler with full C++11 support, MPI (e.g., MPICH, OpenMPI, IntelMPI), FFTW3, GSL, and HDF5. Hence, the corresponding environment modules must be loaded, e.g.,
Building the GADGET code requires a compiler with full C++11 support, MPI (e.g., MPICH, OpenMPI, IntelMPI), HDF5, GSL, and FFTW3. Hence, the corresponding environment modules must be loaded, e.g.,
@@ -32,7 +29,12 @@ or get a cloned repository of the code by using
git clone http://gitlab.mpcdf.mpg.de/vrs/gadget4
```
Source code used in the benchmarks (version of June 22, 2021) [./gadget/4.0/gadget4-benchmarks.tar.gz](./gadget/4.0/gadget4-benchmarks.tar.gz)
In this repository you can download:
- A cloned version of the code (version of June 22, 2021): [gadget4.tar.gz](./gadget/4.0/gadget4.tar.gz)
- The code used in the benchmarks (version of June 22, 2021):[gadget4-benchmarks.tar.gz](./gadget/4.0/gadget4-benchmarks.tar.gz)
### Build the Executable
...
...
@@ -46,15 +48,15 @@ example_ics.tar.gz includes initial conditions that are needed for each of the e
1. After decompressing gadget4.tar.gz go to the master folder named `gadget4`. There are two files that need modification: **Makefile.systype** and **Makefile**.
1. In the **Makefile.systype** select one of the system types by uncommenting the corresponding line or add a line with your system, e.g.,
a) In the **Makefile.systype** select one of the system types by uncommenting the corresponding line or add a line with your system, e.g.,
```
#SYSTYPE="XXX-BBB"
```
where XXX = system name and BBB = whatever you may want to include here, e.g., impi, openmpi, etc.
1. In case you uncommented a line corresponding to your system in the **Makefile.systype** then there is nothing to do in the **Makefile**.
b) In case you uncommented a line corresponding to your system in the **Makefile.systype** then there is nothing to do in the **Makefile**.
1. In case you added a line, say #SYSTYPE="XXX-BBB", into the **Makefile.systype** then you must modify the **Makefile** by adding the following lines in 'define available Systems'
c) In case you added a line, say #SYSTYPE="XXX-BBB", into the **Makefile.systype** then you must modify the **Makefile** by adding the following lines in 'define available Systems'
```
ifeq ($(SYSTYPE),"XXX-BBB")
...
...
@@ -98,16 +100,17 @@ make CONFIG=../gadget4-case-A/Config.sh EXEC=../gadget4-case-A/gadget4-exe
### Mechanics of Running Benchmark
The general way to run the benchmarks, assuming SLURM Resource/Batch Manager is:
1. Set the environment modules (see Build the Executable section)
2. In the folder of the test cases, e.g., gadget4-case-A, adapt the SLURM script and submit it
2. In the folder of the test cases, e.g., `gadget4-case-A`, adapt the SLURM script and submit it
```
sbatch slurm_script.sh
```
where the gadget_script.sh has the form (for a run with 1024 cores):
where the slurm_script.sh has the form (for a run with 1024 cores):
