GADGET simulation code, which is a parallel cosmological N-body and SPH code meant for simulations of cosmic structure formation and calculations relevant for galaxy evolution and galactic dynamics.
Provide the Astrophysical community information on the performance and scalability (weak and strong scaling) of the Gadget-4 code associated to three test cases in PRACE Tier-0 supercomputers (JUWELS, MareNostrum4, IRENE-SKL, and IRENE-KNL) and the ARM64 Mont-Blanc 3 prototype (Dibona).
GADGET-4 supports collisionless simulations and smoothed particle hydrodynamics on massively parallel computers. All communication between concurrent execution processes is done either explicitly by means of the message passing interface (MPI), or implicitly through shared-memory accesses on processes on multi-core nodes. The code is mostly written in ISO C++ (assuming the C++11 standard), and should run on all parallel platforms that support at least MPI-3.
## Characteristics of Benchmark
GADGET-4 supports collisionless simulations and smoothed particle hydrodynamics on massively parallel computers. All communication between concurrent execution processes is done either explicitly by means of the message passing interface (MPI), or implicitly through shared-memory accesses on processes on multi-core nodes. The code is mostly written in ISO C++ (assuming the C++11 standard), and should run on all parallel platforms that support at least MPI-3. So far, the compatibility of the code with current Linux/UNIX-based platforms has been confirmed on a large number of systems.
GADGET-4 was compiled with the optimisation level O3. In order to have a proper scalability analysis the tests we carried out with one MPI-task per core and 16 tasks per CPU. Hence, a total of 32 tasks (cores) dedicated to the calculations were used per compute node. An extra core to handle the MPI communications was used per compute node. The tests were carried out using two modes: Intel and GCC compiled MPI API and libraries (HPF5, GSL, FFTW3).
## Mechanics of Building Benchmark
...
...
@@ -40,32 +42,47 @@ There are two files to obtain from the repository: gadget4.tar.gz and example_ic
gadget4.tar.gz includes the source code, examples, buildsystem, and documentation folders. It also includes Makefile and Makefile.systype (or a template)
example_ics.tar.gz includes initial conditions that are needed for each of the examples. When untarred you generate a folder named ExampleICs. You may download the examples initial conditions from (./gadget/4.0/example_ics.tar.gz)
example_ics.tar.gz includes initial conditions that are needed for each of the examples. When untarred you generate a folder named ExampleICs. You may download the examples initial conditions from (``./gadget/4.0/example_ics.tar.gz``)
1. After decompressing gadget4.tar.gz go to the master folder named gadget4. There are two files that need modification: Makefile.systype and Makefile.
1. After decompressing gadget4.tar.gz go to the master folder named gadget4 and adapt the Makefile.systype file to your needs. That is, select one of the system types by uncommenting the corresponding line or add, e.g.,
- In the Makefile.systype select one of the system types by uncommenting the corresponding line or add a line with your system, e.g.,
```
#SYSTYPE="XXX-impi"
#SYSTYPE="XXX-BBB"
```
where XXX = system name
where XXX = system name and BBB = whatever you may want to include here, e.g., impi, openmpi, etc.
- In case you uncommented a line corresponding to your system in the Makefile.systype then there is nothing to do in the Makefile.
- In case you added a line, say #SYSTYPE="XXX-impi", into the Makefile.systype then you must modify the Makefile by adding the following lines in 'define available Systems'
```
ifeq ($(SYSTYPE),"XXX-BBB")
include buildsystem/Makefile.comp.XXX-BBB
include buildsystem/Makefile.path.XXX-BBB
endif
```
where XXX = system name and BBB = whatever you may want to include here, e.g., impi, openmpi, etc.
2. In the folder buildsystem make sure you have the Makefile.comp.xxx and Makefile.path.xxx (xxx = cluster name)
set with the proper paths and compilation options, respectively.
set with the proper paths and compilation options, respectively. Either chose the existing files or create new ones that reflect your system paths and compiler.
3. The folder examples has several subfolders of test cases. From one of these subfolders, e.g., CollidingGalaxiesSFR, copy Config.sh to the master folder.
4. In the master folder compile the code
4. In the master folder compile the code
```
make CONFIG=Config.sh EXEC=gadget4-exe
```
where EXEC is the name of the executable.
5. Create a folder named Test-Case-A. Copy gadget4-exe to Test-Case-A. From the examples subfolder CollidingGalaxiesSFR copy the files param.txt and TREECOOL to Test-Case-A.
5. Create a folder named Run_CollidingGalaxies. Copy gadget4-exe, and the files param.txt and TREECOOL existing in the subfolder CollidingGalaxiesSFR to Run_CollidingGalaxies.
6. In the folder Test-Case-A modify param.txt to include the adequate path to the initial conditions file ics_collision_g4.dat located in the folder ExampleICs and modify the memory per core to that of the system you are using.
6. In the folder Run_CollidingGalaxies modify param.txt to include the adequate path to the initial conditions file ics_collision_g4.dat located in the folder ExampleICs and modify the memory per core to that of the system you are using.
7. Run the code using mpirun or submit a SLURM script.
