Newer
Older
## Purpose of Benchmark
GADGET simulation code, which is a parallel cosmological N-body and SPH code meant for simulations of cosmic structure formation and calculations relevant for galaxy evolution and galactic dynamics.
## Characteristics of Benchmark
GADGET-4 supports collisionless simulations and smoothed particle hydrodynamics on massively parallel computers. All communication between concurrent execution processes is done either explicitly by means of the message passing interface (MPI), or implicitly through shared-memory accesses on processes on multi-core nodes. The code is mostly written in ISO C++ (assuming the C++11 standard), and should run on all parallel platforms that support at least MPI-3. So far, the compatibility of the code with current Linux/UNIX-based platforms has been confirmed on a large number of systems.
Building the GADGET code requires a compiler with full C++11 support, MPI (e.g., MPICH, OpenMPI, IntelMPI), FFTW3, GSL, and HDF5. Hence, the corresponding environment modules must be loaded, e.g.,
```
module load mpi/OpenMPI/4.0.3-GCC-9.3.0 data/HDF5/1.10.6-gompi-2020a \
numlib/FFTW/3.3.8-gompi-2020a numlib/GSL/2.6-GCC-9.3.0
```
Latest Release can be downloaded from [https://gitlab.mpcdf.mpg.de/vrs/gadget4](https://gitlab.mpcdf.mpg.de/vrs/gadget4)
or get a cloned repository of the code by using
git clone http://gitlab.mpcdf.mpg.de/vrs/gadget4
Source code used in the benchmarks (version of June 21, 2021) [./gadget/4.0/gadget4.tar.gz](./gadget/4.0/gadget4.tar.gz)
There are two files to obtain from the repository: gadget4.tar.gz and example_ics.tar.gz
gadget4.tar.gz includes the source code, examples, buildsystem, and documentation folders. It also includes Makefile and Makefile.systype (or a template)
example_ics.tar.gz includes initial conditions that are needed for each of the examples. When untarred you generate a folder named ExampleICs. You may download the examples initial conditions from (./gadget/4.0/example_ics.tar.gz)
1. After decompressing gadget4.tar.gz go to the master folder named gadget4 and adapt the Makefile.systype file to your needs. That is, select one of the system types by uncommenting the corresponding line or add, e.g.,
2. In the folder buildsystem make sure you have the Makefile.comp.xxx and Makefile.path.xxx (xxx = cluster name)
set with the proper paths and compilation options, respectively.
3. The folder examples has several subfolders of test cases. From one of these subfolders, e.g., CollidingGalaxiesSFR, copy Config.sh to the master folder.
4. In the master folder compile the code
5. Create a folder named Test-Case-A. Copy gadget4-exe to Test-Case-A. From the examples subfolder CollidingGalaxiesSFR copy the files param.txt and TREECOOL to Test-Case-A.
6. In the folder Test-Case-A modify param.txt to include the adequate path to the initial confidions file ics_collision_g4.dat located in the folder ExampleICs and modify the memory per core to that of the system you are using.
7. Run the code using mpirun or submit a SLURM script.
1. Download and untar a test case tarball, e.g., gadget4-caseA.tar.gz (see below) and the gadget code used in the benchmarks. The folder gadget4-caseA has the files Config.sh, ics_collision_g4.dat, param.txt, and TREECOOL. The param.txt file has the path for the initial conditions.
2. Change to the folder named gadget4 and adapt the file Makefile.systype to your needs. Do not forget to adapt also the Makefile, by adding the following lines in 'define available Systems'
```
ifeq ($(SYSTYPE),"oblivion-impi")
include buildsystem/Makefile.comp.XXX-BBB
include buildsystem/Makefile.path.XXX-BBB
endif
```
where XXX = cluster name and BBB= whatever you may want to include here, e.g., impi, openmpi, etc.
3. Compile the code using the Config.sh file in gadget4-caseA
```
make CONFIG=../gadget4-caseA/Config.sh EXEC=../gadget4-caseA/gadget4-exe
```
4. Change to folder gadget4-caseA and make sure that in the file param.txt the memory size per core is the correct one for the system you are using.
5. Run the code directly with mpirun or submit a SLURM script.
### Mechanics of Running Benchmark
The general way to run the benchmarks, assuming SLURM Resource/Batch Manager is:
1. Set the environment modules (see Build the Executable section)
2. In the folder of the test cases, e.g., gadget4-caseA, adapt the SLURM script and submit it
```
sbatch gadget_script.sh
```
where the gadget_script.sh has the form:
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
```
#!/bin/bash -l
#SBATCH --time=04:00:00
#SBATCH --account=astro_00
#SBATCH --job-name=collgal-tMax=1.0-0512
#SBATCH --output=g_collgal_%j.out
#SBATCH --error=g_collgal_%j.error
#SBATCH --nodes=16
#SBATCH --ntasks=512
#SBATCH --cpus-per-task=1
#SBATCH --ntasks-per-socket=16
#SBATCH --exclusive
#SBATCH --partition=debug
echo
echo "Running on hosts: $SLURM_NODELIST"
echo "Running on $SLURM_NNODES nodes."
echo "Running on $SLURM_NPROCS processors."
echo "Current working directory is `pwd`"
echo
srun ./gadget4_collgal param.txt
```
Where:
* gadget4_collgal is the executable.
* param.txt is the input parameter file.
### UEABS Benchmarks
**A) `Colliding galaxies with star formation`**
This simulation with setup in the folder CollidingGalaxiesSFR considers the collision of two compound galaxies made up of a dark matter halo, a stellar disk and bulge, and cold gas in the disk that undergoes star formation. Radiative cooling due to helium and hydrogen is included. Star formation and feedback is modelled with a simple subgrid treatment.
[Download test Case A](./gadget/4.0/gadget4-caseA.tar.gz)
**B) `Cosmological DM-only simulation with IC creation`**
The setup in DM-L50-N128 simulates a small box of comoving side-length 50 Mpc/h using 128^3 dark matter particles. The initial conditions are created on the fly upon start-up of the code, using second order Lagrangian perturbation theory with a starting redshift of z=63. The LEAN option and 32-bit arithmetic are enabled to minimize memory consumption of the code.
Gravity is computed with the TreePM algorithm at expansion order p=3. Three output times are defined, for which FOF group finding is enabled, and power spectra are computed as well for the snapshots that are produced. Also, the code is asked to compute a power spectrum for each output.
[Download test Case B](./gadget/4.0/gadget4-caseB.tar.gz)
**C) `Adiabatic collapse of a gas sphere`**
This simulation in G2-gassphere considers the gravitational collapse of a self-gravitating sphere of gas which initially has a 1/r density profile and a very low temperature. The gas falls under its own weight to the centre, where it bounces back and a strong shock wave that moves outwards develops. The simulation uses Newtonian physics in a natural system of units (G=1).
[Download test Case C](./gadget/4.0/gadget4-caseC.tar.gz)
## Performance
GADGET reports in log file both time and performance.
** `Performance` in `ns/day` units : `grep Performance logfile | awk -F ' ' '{print $2}'`. **
** `Execution Time` in `seconds` : `grep Time: logfile | awk -F ' ' '{print $3}'`**