*Dynamic Sparce Data Exchange* (DSDE), is a problem central to many HPC applicaitons, such as graph computations (e.g. breadth firsrt search), sparse matrix computations with sparsity mutations, and **particle codes**.
It occurs in situations, where:
* Each process has a set of data items to send a small number of other processes.
* The destination processes typically do not know how much they will receive from which other process.
* In addition, the send-to relations are changing dynamically and are somewhat localized.
This code sample demonstrates:
* How to implement DSDE using a **non-blocking barrier**, i.e. `MPI_Ibarrier` which is new in MPI-3, together with non-blocking point-to-point communications.
* How to use MPI-Datatypes to **send and receive structured strided data** directly, avoiding send and receive buffer packing and unpacking.
* How to provide a **custom reduce operation** to `MPI_Reduce`.
* How to use a **cartesian topology** for MPI communication.
The code sample is structured as follows:
*`configuration.c`, `configuration.h`: Command-line parsing and basic logging facilities.
*`main.c`: The main program.
*`mpicomm.c`, `mpicomm.h`: **Probably the most interesting part**, implementing the core message-passing functionality:
*`mpi_communicate_particles_dynamic`: Implementation of DSDE using `MPI_Ibarrier`, `MPI_Issend`, `MPI_IProbe` and `MPI_Recv`.
*`mpi_communicate_particles_collective`: Implementation of DSDE using `MPI_Alltoall` (exchange of message sizes) and `MPI_Alltoallw` (exchange of particle data). Expected to be not as scalable as the first implementation.
*`mpi_reduce_sim_info`: Reduction for simulation statistics (min./max. particles per processor domain, max. velocity, etc.)
*`mpitypes.c`, `mpitypes.h`: Code for initialization of custom MPI-Datatypes.
*`mpitype_part_init_send`: Creates MPI-Datatype for sending structured strided particle data.
*`mpitpye_part_init_recv`: Creates MPI-Datatype for receiving strucutred particle data in contiguous blocks.
*`particles.c`, `particles.h`: Code for maintaining the particle data structure.
*`random.c`, `random.h`: Helper functions for initializing particle data with random values.
*`simulation.c`, `simulation.h`:
## Release Date
2016-01-18
## Version History
* 2016-01-18: Initial Release on PRACE CodeVault repository
## Contributors
* Thomas Ponweiser - [thomas.ponweiser@risc-software.at](mailto:thomas.ponweiser@risc-software.at)
## Copyright
This code is available under APACHE-2 Licence, see also the license file in the CodeVault root directory.
## Languages
This sample is entirely written in C.
## Parallelisation
This sample uses MPI-3 for parallelisation.
## Level of the code sample complexity
Intermediate / Advanced
## Compiling
In order to build the sample, only a working MPI implementation supporting MPI-3 must be available. To compile, simply run:
make
If you need to use a MPI wrapper compiler other than `mpicc`, e.g. `mpiicc`, type:
make MPICC=mpiicc
In order to specify further compilation flags, e.g. `-g`, type:
make CFLAGS="-g"
## Running
To run the program, use something similar to
mpirun -n [nprocs] ./dsde -x [nx] -y [ny]
either on the command line or in your batch script, where `nx` and `ny` specify the number of cells in a two-dimensional grid in x- and y-direction respectively such that `nx`*`ny` = `nprocs`. If one of `-x` or `-y` is ommitted, `ny` or `nx` is determined automatically.
### Further Command line arguments
*`-v [0-3]`: Specify the output verbosity level - 0: OFF; 1: INFO (Default); 2: DEBUG; 3: TRACE;
*`-d [rank]`: Debug MPI process with specified rank. Enables debug output for the specified rank (otherwise only output of rank 0 is written) and, if compiled with `-CFLAGS="DDEBUG_ATTACH"`, enables a waiting loop for the specified rank which allows to attach a debugger.
*`--use-cart-topo`: Use MPI Communicator with cartesian topology (according to `nx` and `ny`) for all MPI communications. If supported, this allows MPI a better mapping of ranks to the physical hardware topology.
*`--collective`: Use collective operations for DSDE, i.e. `MPI_Alltoall`for exchanging message sizes and `MPI_Alltoallw` for exchanging particle data.
*`--dynamic`: Use more sophisticated communication scheme for DSDE, i.e. `MPI_Issend`, `MPI_IProbe`, `MPI_Ibarrier`.
*`-x [nx]`: Number of cells (processes) in x-direction.
*`-y [ny]`: Number of cells (processes) in y-direction.
*`-i [iterations]`*: Number of iterations for the particle simulation; Default: 100.
*`-n [particles-per-cell]`*: Number of particles per cell or process (initial); Default: 16k (= 16 * 1024).
*`-N [particles-total]`*: Number of particles in total (handy for strong-scalability analyses).
*`-V [max-velocity]`: Maximum magnitude of initial particle velocity; Default: 1 (= 1 cell width / time unit).
**`-t [delta-t]`: Time delta for simulation step; Default: 1.
*`-M [max-mass]`: Maximum particle mass; Default: 1.
*`-F [max-random-force]`: Maximum magnitude of random force applied to each particle in each iteration; Default: 0 (Disabled).
Large numbers as arguments to the options `-i`, `-n` or `-N` can be suffixed with 'k' or 'M'. For example, `-n 16k` specifies 16 * 1024 particles per cell; `-N 1M` specifies 1024 * 1024 (~1 million) particles in total.
