# README - Dynamic Sparse Data Exchange Example


## Description

*Dynamic Sparse Data Exchange* (DSDE) is a problem central to many HPC applicaitons, such as graph computations (e.g. breadth firsrt search), sparse matrix computations with sparsity mutations and **particle codes**.

It occurs in situations where:

 * Each process has a set of data items to send a small number of other processes.
 * The destination processes typically do not know how much they will receive from which other process.
 * In addition, the send-to relations are changing dynamically and are somewhat localized.

This code sample demonstrates:

 * Three different approaches for implementing DSDE:
   1. Using **remote memory access** (a.k.a. **one-sided communication**), i.e. `MPI_Accumulate`
   1. Using a **non-blocking barrier**, i.e. `MPI_Ibarrier` (introduced in MPI-3)
   1. Using **collective communication**, i.e. `MPI_Alltoall` and `MPI_Alltoallw` (not as scalable as the first two approaches).
 * How to use MPI-Datatypes to **send and receive non-contiguous structured data** directly, avoiding send and receive buffer packing and unpacking.
 * How to provide a **custom reduce operation** to `MPI_Reduce`.
 * How to use a **cartesian topology** for MPI communication.

The code sample is structured as follows:

 * `configuration.c`, `configuration.h`: Command-line parsing and basic logging facilities.
 * `main.c`: The main program.
 * `mpicomm.c`, `mpicomm.h`: **Probably the most interesting part**, implementing the core message-passing functionality:
   * `mpi_communicate_particles_rma`: Implementation of DSDE using `MPI_Accumulate`, `MPI_Isend`, `MPI_Probe` and `MPI_Recv`.
   * `mpi_communicate_particles_dynamic`: Implementation of DSDE using `MPI_Ibarrier`, `MPI_Issend`, `MPI_IProbe` and `MPI_Recv`.
   * `mpi_communicate_particles_collective`: Alternative implementation of DSDE using `MPI_Alltoall` and `MPI_Alltoallw`.
   * `mpi_reduce_sim_info`: Reduction for simulation statistics (minimum/maximum particles per processor domain, maximum velocity, etc.)
 * `mpitypes.c`, `mpitypes.h`: Code for initialization of custom MPI-Datatypes.
   * `mpitype_part_init_send`: Creates MPI-Datatype for sending non-contiguous structured particle data.
   * `mpitpye_part_init_recv`: Creates MPI-Datatype for receiving structured particle data in contiguous blocks.
 * `particles.c`, `particles.h`: Code for maintaining the particle data structure.
 * `random.c`, `random.h`: Helper functions for initializing particle data with random values.
 * `simulation.c`, `simulation.h`: Demo implementation of particle simulation.
 * `vector.c`, `vector.h`: Helper data structure for arrays of 2-dimensional vectors.


## Release Date

2016-01-18


## Version History

 * 2016-01-18: Initial Release on PRACE CodeVault repository


## Contributors

 * Thomas Ponweiser - [thomas.ponweiser@risc-software.at](mailto:thomas.ponweiser@risc-software.at)


## Copyright

This code is available under Apache License, Version 2.0 - see also the license file in the CodeVault root directory.


## Languages

This sample is entirely written in C.


## Parallelisation

This sample uses MPI-3 for parallelisation.


## Level of the code sample complexity

Intermediate / Advanced


## Compiling

Follow the compilation instructions given in the main directory of the kernel samples directory (`/hpc_kernel_samples/README.md`).

## Running

To run the program, use something similar to

    mpirun -n [nprocs] ./4_nbody_dsde -x [nx] -y [ny]

either on the command line or in your batch script, where `nx` and `ny` specify the number of cells in a two-dimensional grid in x- and y-direction respectively such that `nx` * `ny` = `nprocs`. If one of `-x` or `-y` is ommitted, `ny` or `nx` is determined automatically.


### Command line arguments

 * `-v [0-3]`: Specify the output verbosity level - 0: OFF; 1: INFO (Default); 2: DEBUG; 3: TRACE;
 * `-g [rank]`: Debug MPI process with specified rank. Enables debug output for the specified rank (otherwise only output of rank 0 is written) and, if compiled with `-CFLAGS="-g -DDEBUG_ATTACH"`, enables a waiting loop for the specified rank which allows to attach a debugger.
 * `--use-cart-topo`: Use MPI Communicator with Cartesian topology (according to `nx` and `ny`) for all MPI communications. If supported, this allows MPI to reorder the processes in order to choose a good embedding of the virtual topology to the physical machine.
 * `--rma`: Use remote memory access for DSDE, i.e. `MPI_Accumulate`, for exchanging message sizes and `MPI_Isend`, `MPI_Probe`, `MPI_Recv` for exchanging particle data (Default).
 * `--dynamic`: Use non-blocking barrier for DSDE, i.e. `MPI_Issend`, `MPI_IProbe`, `MPI_Ibarrier`.
 * `--collective`: Use collective operations for DSDE, i.e. `MPI_Alltoall` for exchanging message sizes and `MPI_Alltoallw` for exchanging particle data.
 * `-x [nx]`: Number of cells (processes) in x-direction.
 * `-y [ny]`: Number of cells (processes) in y-direction.
 * `-i [iterations]`: Number of iterations for the particle simulation; Default: 100.
 * `-n [particles-per-cell]`: Number of particles per cell or process (initial); Default: 16k (= 16 * 1024).
 * `-N [particles-total]`: Number of particles in total (handy for strong-scalability analyses).
 * `-V [max-velocity]`: Maximum magnitude of initial particle velocity; Default: 1 (= 1 cell width / time unit).
 *  `-t [delta-t]`: Time delta for simulation step; Default: 1.
 * `-m [min-mass]`: Minimum particle mass; Default: 1.
 * `-M [max-mass]`: Maximum particle mass; Default: 1.
 * `-F [max-random-force]`: Maximum magnitude of random force applied to each particle in each iteration; Default: 0 (Disabled).

For large numbers as arguments to the options `-i`, `-n` or `-N`, the suffixes 'k' or 'M' may be used. For example, `-n 16k` specifies 16 * 1024 particles per cell; `-N 1M` specifies 1024 * 1024 (~1 million) particles in total.


### Example

If you run

    mpirun -n 16 ./4_nbody_dsde -x 4 -N 1M --collective --use-cart-topo

the output should look similar to

    Configuration:
     * Verbosity level:         INFO (1)
     * Use MPI Cart. Topology:  YES
     * Transmission mode:       Collective - MPI_Alltoall / MPI_Alltoallw
     * Number of cells:         4 x 4
     * Particles per cell:      65536
     * Init. Velocity max:      1.000000
     * Timestep:                1.000000
     * Particle mass (min-max): 1.000000 - 1.000000
     * Max. random force:       0.000000

    INFO: MPI reordered ranks: NO

    Simulation status (initial):
     * Number of particles (total):  1048576
     * Particles per cell (min-max): 65536 - 65536
     * Max. velocity:                1.000000

    Simulation satus (final):
     * Number of particles (total):  1048576
     * Particles per cell (min-max): 65111 - 66072
     * Max. velocity:                1.000000

## Benchmarks

### Strong scaling and communication comparison on Supermuc

Strong scaling test on Supermuc with 20M particles with different number of cores and different communication modes.

Hardware: Supermuc Phase 1 Thin Nodes. Intel Sandy Bridge-EP Xeon E5-2680 8C @2.7GHz, 16 Cores per node.
MPI: Intel MPI 5.1.3.181

Commandline: 

	mpiexec -n {ranks} ./4_nbody_dsde -x {x} -N 20M -i 8000 {flags}
	
where `{ranks}` is the number of cores, `{flags}` is [`rma`, `dynamic`, `collective`] and the number of cells `{x}` = `log2(ranks)/2` for reasons of balancing the data.
As Supermuc Phase 1 Thin nodes are 8 core dual socket systems, 16 tasks per node are used.

![chart](hpc_kernel_samples/n-body_methods/dynamic_sparse_data_exchange/benchmarks/SupermucChart.PNG)
