diff --git a/dynamic_sparse_data_exchange/README.md b/dynamic_sparse_data_exchange/README.md new file mode 100644 index 0000000000000000000000000000000000000000..34056705237d092310442a6568dd2b79a3180c2b --- /dev/null +++ b/dynamic_sparse_data_exchange/README.md @@ -0,0 +1,122 @@ +# README - Dynamic Sparse Data Exchange Example + +## Description +*Dynamic Sparce Data Exchange* (DSDE), is a problem central to many HPC applicaitons, such as graph computations (e.g. breadth firsrt search), sparse matrix computations with sparsity mutations, and **particle codes**. + +It occurs in situations, where: + * Each process has a set of data items to send a small number of other processes. + * The destination processes typically do not know how much they will receive from which other process. + * In addition, the send-to relations are changing dynamically and are somewhat localized. + +This code sample demonstrates: + * How to implement DSDE using a **non-blocking barrier**, i.e. `MPI_Ibarrier` which is new in MPI-3, together with non-blocking point-to-point communications. + * How to use MPI-Datatypes to **send and receive structured strided data** directly, avoiding send and receive buffer packing and unpacking. + * How to provide a **custom reduce operation** to `MPI_Reduce`. + * How to use a **cartesian topology** for MPI communication. + +The code sample is structured as follows: + * `configuration.c`, `configuration.h`: Command-line parsing and basic logging facilities. + * `main.c`: The main program. + * `mpicomm.c`, `mpicomm.h`: **Probably the most interesting part**, implementing the core message-passing functionality: + * `mpi_communicate_particles_dynamic`: Implementation of DSDE using `MPI_Ibarrier`, `MPI_Issend`, `MPI_IProbe` and `MPI_Recv`. + * `mpi_communicate_particles_collective`: Implementation of DSDE using `MPI_Alltoall` (exchange of message sizes) and `MPI_Alltoallw` (exchange of particle data). Expected to be not as scalable as the first implementation. + * `mpi_reduce_sim_info`: Reduction for simulation statistics (min./max. particles per processor domain, max. velocity, etc.) + * `mpitypes.c`, `mpitypes.h`: Code for initialization of custom MPI-Datatypes. + * `mpitype_part_init_send`: Creates MPI-Datatype for sending structured strided particle data. + * `mpitpye_part_init_recv`: Creates MPI-Datatype for receiving strucutred particle data in contiguous blocks. + * `particles.c`, `particles.h`: Code for maintaining the particle data structure. + * `random.c`, `random.h`: Helper functions for initializing particle data with random values. + * `simulation.c`, `simulation.h`: + +## Release Date +2016-01-18 + +## Version History + * 2016-01-18: Initial Release on PRACE CodeVault repository + +## Contributors + * Thomas Ponweiser - [thomas.ponweiser@risc-software.at](mailto:thomas.ponweiser@risc-software.at) + +## Copyright +This code is available under APACHE-2 Licence, see also the license file in the CodeVault root directory. + +## Languages +This sample is entirely written in C. + +## Parallelisation +This sample uses MPI-3 for parallelisation. + +## Level of the code sample complexity +Intermediate / Advanced + +## Compiling +In order to build the sample, only a working MPI implementation supporting MPI-3 must be available. To compile, simply run: + + make + +If you need to use a MPI wrapper compiler other than `mpicc`, e.g. `mpiicc`, type: + + make MPICC=mpiicc + +In order to specify further compilation flags, e.g. `-g`, type: + + make CFLAGS="-g" + +## Running +To run the program, use something similar to + + mpirun -n [nprocs] ./dsde -x [nx] -y [ny] + +either on the command line or in your batch script, where `nx` and `ny` specify the number of cells in a two-dimensional grid in x- and y-direction respectively such that `nx` * `ny` = `nprocs`. If one of `-x` or `-y` is ommitted, `ny` or `nx` is determined automatically. + +### Further Command line arguments + * `-v [0-3]`: Specify the output verbosity level - 0: OFF; 1: INFO (Default); 2: DEBUG; 3: TRACE; + * `-d [rank]`: Debug MPI process with specified rank. Enables debug output for the specified rank (otherwise only output of rank 0 is written) and, if compiled with `-CFLAGS="DDEBUG_ATTACH"`, enables a waiting loop for the specified rank which allows to attach a debugger. + * `--use-cart-topo`: Use MPI Communicator with cartesian topology (according to `nx` and `ny`) for all MPI communications. If supported, this allows MPI a better mapping of ranks to the physical hardware topology. + * `--collective`: Use collective operations for DSDE, i.e. `MPI_Alltoall`for exchanging message sizes and `MPI_Alltoallw` for exchanging particle data. + * `--dynamic`: Use more sophisticated communication scheme for DSDE, i.e. `MPI_Issend`, `MPI_IProbe`, `MPI_Ibarrier`. + * `-x [nx]`: Number of cells (processes) in x-direction. + * `-y [ny]`: Number of cells (processes) in y-direction. + * `-i [iterations]`*: Number of iterations for the particle simulation; Default: 100. + * `-n [particles-per-cell]`*: Number of particles per cell or process (initial); Default: 16k (= 16 * 1024). + * `-N [particles-total]`*: Number of particles in total (handy for strong-scalability analyses). + * `-V [max-velocity]`: Maximum magnitude of initial particle velocity; Default: 1 (= 1 cell width / time unit). + * * `-t [delta-t]`: Time delta for simulation step; Default: 1. + * `-m [min-mass]`: Minimum particle mass; Default: 1. + * `-M [max-mass]`: Maximum particle mass; Default: 1. + * `-F [max-random-force]`: Maximum magnitude of random force applied to each particle in each iteration; Default: 0 (Disabled). + +Large numbers as arguments to the options `-i`, `-n` or `-N` can be suffixed with 'k' or 'M'. For example, `-n 16k` specifies 16 * 1024 particles per cell; `-N 1M` specifies 1024 * 1024 (~1 million) particles in total. + +### Example + +If you run + + mpirun -n 16 ./dsde -x 4 -N 1M --collective --use-cart-topo + +the command line output should look similar to + +``` +Configuration: + * Verbosity level: INFO (1) + * Use MPI Cart. Topology: YES + * Transmission mode: Collective - MPI_Alltoall / MPI_Alltoallw + * Number of cells: 4 x 4 + * Particles per cell: 65536 + * Init. Velocity max: 1.000000 + * Timestep: 1.000000 + * Particle mass (min-max): 1.000000 - 1.000000 + * Max. random force: 0.000000 + +INFO: MPI reordered ranks: NO + +Simulation status (initial): + * Number of particles (total): 1048576 + * Particles per cell (min-max): 65536 - 65536 + * Max. velocity: 1.000000 + +Simulation satus (final): + * Number of particles (total): 1048576 + * Particles per cell (min-max): 65111 - 66072 + * Max. velocity: 1.000000 +```