.. _4ip-extension:

4IP-extension: Run on PCP machines
==================================

This document will describe the work done under the PRACE-4IP extension. This task is dedicated to provide useful information on application *performance and energy usage* on next generation systems on the path towards exacsale. It will be caried out running the accelerated UEABS on PCP systems to obtain energy metrics on *OpenPower+GPU, Xeon Phi and FPGA*.

Organisation is publicly available to anyone interrested in the project and rely on the following tools:
 * a slack channel for chat purpose: `PRACE PCP slack channel`_ , ask `Victor Cameo Ponz`_ for registration link.
 * a mailing list: prace-4ip-wp7.extension@fz-juelich.de, subscribe here: `mailing list registration page`_.
 * this documentation.

The following methodology will be followed to lead this task:
 #. white paper indicating what applications will be run on which prototypes (meeting the PRACE milestone **MS33 due by August 2017/M4**)
 #. grant access to machines (cut off September 2017 )
 #. run benchmarks (cut off October/November)
 #. gather results and report *Applications Performance and Energy Usage* (this adresses the PRACE deliverable **D7.7 due by December 2017/M8**)

Code definition
---------------

The table :ref:`table-code-definition` shows all codes availabe with UEABS (regular and accelerated). It states for each codes, tageted architechures and BCOs. Note that due to tight deadlines, efforts to port codes to new architechures will have to be minimal.

.. _table-code-definition:
.. table:: Code definition
   :widths: auto

   +------------------------+--------------------------------+-----------------------------+
   |                        |           Will run on          |                             |
   |   Code name            +--------------+----------+------+   4IP-extension BCO         +
   |                        | Power8 + GPU | Xeon Phi | FPGA |                             |
   +========================+==============+==========+======+=============================+
   | ALYA                   | ✓ (?)        | ✓ (?)    | ✗    | ? Ricard Borrell (BSC) ??   |
   +------------------------+--------------+----------+------+-----------------------------+
   | Code_Saturne           | ✓            | ✓        | ✗    | Charles Moulinec (STFC)     |
   +------------------------+--------------+----------+------+-----------------------------+
   | CP2K                   | ✓            | ✓        | ✗    | Arno Proeme (EPCC)          |
   +------------------------+--------------+----------+------+-----------------------------+
   | GADGET                 | ✗            | ✓        | ✗    | Volker Weinberg (LRZ)       |
   +------------------------+--------------+----------+------+-----------------------------+
   | GENE                   |              |          | ✗    |  ?                          |
   +------------------------+--------------+----------+------+-----------------------------+
   | GPAW                   |  ? (✗)       | ✓        | ✗    | Martti Louhivuori (CINCA)   |
   +------------------------+--------------+----------+------+-----------------------------+
   | GROMACS                |  ✓           | ✓        | ✗    | Dimitris Dellis (GRNET)     |
   +------------------------+--------------+----------+------+-----------------------------+
   | NAMD                   |  ✓           | ✓        | ✗    | Dimitris Dellis (GRNET)     |
   +------------------------+--------------+----------+------+-----------------------------+
   | NEMO                   |              |          | ?    |  ?                          |
   +------------------------+--------------+----------+------+-----------------------------+
   | PFARM                  |              |          | ✗    |  ?                          |
   +------------------------+--------------+----------+------+-----------------------------+
   | QCD                    |    ✓         | ✓        | ✗    | Arno Proeme (EPCC)          |
   +------------------------+--------------+----------+------+-----------------------------+
   | Quantum Espresso       |    ✓         | ? (✓)    | ?    | Andrew Emerson (CINECA)     |
   +------------------------+--------------+----------+------+-----------------------------+
   | SHOC                   |    ✓         | ✗        | ? (✓)| Valeriu Codreanu (SurfSARA) |
   +------------------------+--------------+----------+------+-----------------------------+
   | Specfem3D_Globe        |    ✓         | ✓        | ?    | Victor Cameo Ponz (CINES)   |
   +------------------------+--------------+----------+------+-----------------------------+


PCP systems description and registration process
------------------------------------------------

This section describes the systems where BCOs have been granted access.
The table :ref:`table-pcp-systems` sums up the when registration process starts:

.. _table-pcp-systems:
.. table:: PCP Systems
   :widths: auto

   +--------------+--------------+----------------------------+---------------+-----------------------------+
   |  Technology  | Theoretical  | Manufacturer               | Host          | Availability                |
   |              | peak perf    |                            |               |                             |
   +==============+==============+============================+===============+=============================+
   | Power8 + GPU | 877 TFlop/s  | `E4 computer engineering`_ | CINECA_ (It)  | Start migration of nodes to |
   |              |              |                            |               | CINECA in June/July 2017    |
   |              |              |                            |               | **shifted to midSeptember** |
   +--------------+--------------+----------------------------+---------------+-----------------------------+
   | Xeon Phi     | 512 TFlop/s  | `Atos/Bull`_               | CINES_ (Fr)   | June 2017 (now availabe)    |
   +--------------+--------------+----------------------------+---------------+-----------------------------+
   | FPGA         | N/A          | MAXELER_                   | JSC_ (De)     | August 2017                 |
   |              |              |                            |               | **shifted to midSeptember** |
   +--------------+--------------+----------------------------+---------------+-----------------------------+

.. note:: The registration process included whithin is aimed at BCOs only.


Power8 + GPU
^^^^^^^^^^^^

This machine has been designed by `E4 computer engineering`_ and is hosted at CINECA_ in Bologna, Italy.

.. note:: In order to access the machine BCO should register on the `CINECA user datatabase`_ and ask `Victor Cameo Ponz`_ to be added to the 4IP-extension project.

Compute technology
""""""""""""""""""

Hardware features fat-nodes with the following design:
 * 45 nodes with x2 IBM POWER8 processors and x4 NVIDIA P100 GPU
 * intranode comunications integrated using NVLink
 * extranode comunications integrated using Infiniband ERD interconnect
 * CPU and GPU liquid cooling based on CoolIT_ solution

Energy sampling technology
""""""""""""""""""""""""""

Xeon Phi
^^^^^^^^

This machine has been designed by `Atos/Bull`_ and is hosted at CINES_ in Montpellier, France.

.. note:: In order to access the machine BCO should fill the request to open login form (`odt <cines-login-form-odt_>`_ or `rtf <cines-login-form-rtf_>`_) and send it back to `Victor Cameo Ponz`_.

Compute technology
""""""""""""""""""
Hardware features the following nodes:
 * 168 nodes with 1x Intel Xeon Phi 7250 processor (KNL)
 * Liquid cooled nodes and PSU
 * MooseFS I/O

Energy sampling technology
""""""""""""""""""""""""""


FPGA
^^^^

This machine has been designed by MAXELER_ and is hosted at JSC_ in Julich, Germany.

Compute technology
""""""""""""""""""

This small pilot system features:
 - 4 MPC-H servers including 2x MAX5 DFE and 2x Intel Xeon processors

Energy sampling technology
""""""""""""""""""""""""""



.. _Victor Cameo Ponz: cameo+4ip-extension@cines.fr
.. _mailing list registration page: https://lists.fz-juelich.de/mailman/listinfo/prace-4ip-wp7.extension

.. _PRACE PCP slack channel: https://prace-pcp.slack.com

.. _E4 computer engineering: https://www.e4company.com
.. _Atos/Bull: https://bull.com/
.. _MAXELER: http://maxeler.com/

.. _CINECA: http://hpc.cineca.it/
.. _CINES: https://www.cines.fr/
.. _JSC: http://www.fz-juelich.de/ias/jsc/EN/Home/home_node.html

.. _CINECA user datatabase: https://userdb.hpc.cineca.it/
.. _cines-login-form-odt: https://www.cines.fr/wp-content/uploads/2014/01/opening_renewal_login_2017.odt
.. _cines-login-form-rtf: https://www.cines.fr/wp-content/uploads/2014/01/opening_renewal_login_2017.rtf

.. _CoolIT: https://www.coolitsystems.com/
.. _Slurm: https://slurm.schedmd.com/
