Skip to content
GitLab
Commit ccd3a7e1 authored by Kurt Lust's avatar Kurt Lust
Browse files

Added build instructions for more PRACE machines, corrected a link that has... 

Added build instructions for more PRACE machines, corrected a link that has changed since the last review.
parent 38aa27ce
Expand all Show whitespace changes
Inline Side-by-side
gpaw/build/examples/LRZ-SuperMUCng-skylake/README.md 0 → 100644
View file @ ccd3a7e1
# Build issues at LRZ
## Compilers used for testing
The following compilers, available on the system at the time of testing and installed
with Spack, were used:
* Intel C/C++/Fortran version 19.1.2.254 (as provided by the intel/2020.0.2 module)
* Intel MPI version 2019.8.254
* Intel MKL 2020.1.217
These are components from the Intel compilers version 2020 update 2.
## Issues
* It is impossible to download files from this system, so all sources have to be
put in place manually before starting the compile. The scripts will generate the
necessary URLs.
* A general issues: At some patch releases of Python 3.8 and 3.9 changes were made
to certain components of the code that installs additional packages. This causes
failures in the installation process of NumPy and SciPy. So it is better to stick
to the versions used in these example scripts (Python 3.8.7 and 3.9.4).
* GPAW tried to compile a file that includes an MPI header with the regular C compiler.
The solution was to set the regular C compiler to the MPI wrapper compiler script.
That issue does not occur on all systems since some systems have the MPI header
files included in ``CFILES`` so that the regular C compile also finds those header
files.
* We failed to get the combination Python 3.9.4 + NumPy 1.19.5 + SciPy 1.5.4 to compile.
It failed during the installation of SciPy, even though this combination worked on
several other systems.
* The system uses TCL modules 4.6 at the time of testing. We failed to get the right
compiler (which were not the default on the system) to load in the module file
unless we first manually unloaded intel-mkl, intel-mpi and intel.
There are no examples on how to do this in the existing module files of the system
which may be due to the fact that spack with rpath is being used.
It might have been a better solution to simply unload spack and use the other
compiler modules that are not spack-managed instead rather than a variant of the
default environment as presented by the system.
gpaw/build/examples/README.md
View file @ ccd3a7e1
# Example build scripts
## BSC-MareNostrum4-skylake
MareNostrum 4 is a Tier-0 cluster installed at the Barcelona Supercomputing
Center. GPAW was installed on the general-purpose block of the cluster,
consisting of 3,456 nodes with dual 24-core Intel Xeon Platinum 8160
"Skylake" CPUs and 96 GB of RAM in most nodes. The interconnect is
Omni-Path.
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](BSC-MareNostrum4-skylake)
## HLRS-Hawk-rome
Hawk is a Tier-0 cluster hosted by HLRS in Germany. The cluster consists of
5,632 compute nodes with dual 64-core AMD EPYC 7742 "Rome" CPUs and 256 GB of
memory. The interconnect is HDR200 InfiniBand in a 9-dimensional hypercube topology.
Most software on the cluster is installed through [Spack](https://spack.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](HLRS-Hawk-rome)
## JSC-JUWELS-skylake
JUWELS is a Tier-0 system hosted by the Jülich Supercomputing Centre. GPAW was only
installed on the so-called "Cluster Module" consisting of 2,271 compute nodes
with dual 24-core Intel Xeon Platinum 8168 "Skylake" CPUs and 96 GB RAM per node.
The interconnect is EDR InfiniBand.
Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](JSC-JUWELS-skylake)
## LRZ-SuperMUCng-skylake
SuperMUC-NG is a Tier-0 system hosted by LRZ in Germany. The cluster has 6,480
compute nodes with dual 24-core Intel Xeon Platinum 8174 CPUs. Most nodes have 96 GB
of RAM. The interconnect is Omni-Path. The cluster runs a SUSE Linux variant.
Most software on the cluster is installed through [Spack](https://spack.io/).
Environment modules are provided through the TCL-based [Environment Modules
package](https://modules.readthedocs.io/).
[Go to the directory](LRZ-SuperMUCng-skylake)
## TGCC-Irene-rome
Joliot-Curie/Irene is a Tier-0 machine hosted at the TGCC in France. GPAW was installed
on the AMD Rome partition. This partition has 2,292 dual 64-core AMD EPYC 7H12 "Rome"
CPUs with 256 GB RAM each. The interconnect is HDR100 InfiniBand.
Environment modules are provided through the TCL-based [Environment Modules
package](https://modules.readthedocs.io/).
[Go to the directory](TGCC-Irene-rome)
## CalcUA-vaughan-rome
Vaughan is a cluster installed at the University of Antwerpen
Vaughan is a cluster installed at the University of Antwerp
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256GB of RAM memory
The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256 GB of RAM memory
per node. Nodes are interconnected through an InfiniBand HDR100 network. The system
does not contain accelerators. The cluster uses CentOS 8 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](CalcUA-vaughan-rome)
......@@ -17,50 +80,30 @@ Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/l
## CalcUA-leibniz-broadwell
Leibniz is a cluster intalled at the University of Antwerpen
Leibniz is a cluster installed at the University of Antwerp
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 153 regular CPU nodes with dual Intel Xeon E5-2680v4
"broadwell" CPUs with 128 or 256 GB of RAM memory per node.
Nodes are interconnected through an Infiniband EDR network.
Nodes are interconnected through an InfiniBand EDR network.
The system has two nodes with dual NVIDIA P100 GPUs and a node
with dual NEC Aurora TSUBASA first generation vector boards.
During the testing period the cluster used CentOS 7 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Most software on the cluster is installed through [EasyBuild](https://easybuild.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](CalcUA-leibniz-broadwell)
## davide @ Cineca
Build instructions for GPU and CPU on the DAVIDE OpenPower GPU cluster @ Cineca.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
[Go to the directory](davide)
## juwels @ Jülich Supercomputing Centre
Build instructions for the CPU nodes of JUWELS.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
[Go to the directory](juwels)
## piz-daint @ CSCS
Build instructions for the Piz Daint (GPU cluster).
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
based on distutils rather than setuptools that was dropped towards the end of
the development of this version of the benchmark suite. They are here because
it is the only still relevant set of instructions for a GPU-based machine.
[Go to the directory](piz-daint)
gpaw/build/examples/davide/gpaw-cpu/build.sh deleted 100644 → 0
View file @ 38aa27ce
### GPAW installation script for D.A.V.I.D.E
# version numbers (modify if needed)
gpaw_version=1.1.0
# installation directory (modify!)
tgt=$CINECA_SCRATCH/lib/gpaw-${gpaw_version}
# setup build environment
module load cuda/9.2.88
module load cudnn/7.1.4--cuda--9.2.88
module load gnu/6.4.0
module load openmpi/3.1.0--gnu--6.4.0
source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
source $CINECA_SCRATCH/lib/python-2018-12-cuda/load.sh 2016-06
source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
export CFLAGS=""
# gpaw
git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
cd gpaw-$gpaw_version
git checkout $gpaw_version
patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
cp ../setup/customize-power8.py .
python setup.py install --customize=customize-power8.py --prefix=$tgt 2>&1 | tee loki-inst
cd ..
sed -e "s|<BASE>|$tgt|g" -e "s|<PYTHONHOME>|$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
# fix permissions
chmod -R g+rwX $tgt
chmod -R o+rX $tgt
gpaw/build/examples/davide/gpaw-cpu/setup/customize-power8.py deleted 100644 → 0
View file @ 38aa27ce
# Setup customisation for gpaw/cuda
import os
# compiler and linker
compiler = 'gcc'
mpicompiler = 'mpicc'
mpilinker = 'mpicc'
extra_compile_args = ['-std=c99', '-mcpu=power8']
# libraries
libraries = ['z']
# openblas
library_dirs += [os.environ['OPENBLAS_ROOT'] + '/lib']
include_dirs += [os.environ['OPENBLAS_ROOT'] + '/include']
libraries += ['openblas']
# scalapack
library_dirs += [os.environ['SCALAPACK_ROOT'] + '/lib']
libraries += ['scalapack']
# libxc
library_dirs += [os.environ['LIBXCDIR'] + '/lib']
include_dirs += [os.environ['LIBXCDIR'] + '/include']
libraries += ['xc']
# GPAW defines
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]
define_macros += [("GPAW_MPI2",1)]
define_macros += [('GPAW_CUDA', '1')]
# ScaLAPACK
scalapack = True
# HDF5
hdf5 = False
gpaw/build/examples/davide/gpaw-cpu/setup/load-gpaw.sh deleted 100644 → 0
View file @ 38aa27ce
#!/bin/bash
module load cuda/9.2.88
module load cudnn/7.1.4--cuda--9.2.88
module load gnu/6.4.0
module load openmpi/3.1.0--gnu--6.4.0
source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
source <PYTHONHOME>/load.sh
source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
export PATH=<BASE>/bin:$PATH
export PYTHONPATH=<BASE>/lib/python2.7/site-packages:$PYTHONPATH
gpaw/build/examples/davide/gpaw/build.sh deleted 100644 → 0
View file @ 38aa27ce
### GPAW installation script for D.A.V.I.D.E
# version numbers (modify if needed)
gpaw_version=cuda
# installation directory (modify!)
tgt=$CINECA_SCRATCH/lib/gpaw-${gpaw_version}
# setup build environment
module load cuda/9.2.88
module load cudnn/7.1.4--cuda--9.2.88
module load gnu/6.4.0
module load openmpi/3.1.0--gnu--6.4.0
source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
source $CINECA_SCRATCH/lib/python-2018-12-cuda/load.sh 2016-06
source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
export CFLAGS=""
# gpaw
git clone https://gitlab.com/mlouhivu/gpaw.git gpaw-$gpaw_version
cd gpaw-$gpaw_version
git checkout $gpaw_version
patch gpaw/eigensolvers/rmm_diis.py ../setup/patch-rmmdiis.diff
cp ../setup/customize-cuda.py .
cd c/cuda
cp ../../../setup/make.inc .
make 2>&1 | tee loki-make
cd -
python setup.py install --customize=customize-cuda.py --prefix=$tgt 2>&1 | tee loki-inst
cd ..
sed -e "s|<BASE>|$tgt|g" -e "s|<PYTHONHOME>|$PYTHONHOME|" setup/load-gpaw.sh > $tgt/load.sh
# fix permissions
chmod -R g+rwX $tgt
chmod -R o+rX $tgt
gpaw/build/examples/davide/gpaw/setup/customize-cuda.py deleted 100644 → 0
View file @ 38aa27ce
# Setup customisation for gpaw/cuda
import os
# compiler and linker
compiler = 'gcc'
mpicompiler = 'mpicc'
mpilinker = 'mpicc'
extra_compile_args = ['-std=c99', '-mcpu=power8']
# libraries
libraries = ['z']
# cuda
library_dirs += [os.environ['CUDA_LIB'], './c/cuda']
include_dirs += [os.environ['CUDA_HOME'] + '/include']
libraries += ['gpaw-cuda', 'cublas', 'cudart', 'stdc++']
# openblas
library_dirs += [os.environ['OPENBLAS_ROOT'] + '/lib']
include_dirs += [os.environ['OPENBLAS_ROOT'] + '/include']
libraries += ['openblas']
# scalapack
library_dirs += [os.environ['SCALAPACK_ROOT'] + '/lib']
libraries += ['scalapack']
# libxc
library_dirs += [os.environ['LIBXCDIR'] + '/lib']
include_dirs += [os.environ['LIBXCDIR'] + '/include']
libraries += ['xc']
# GPAW defines
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]
define_macros += [("GPAW_MPI2",1)]
define_macros += [('GPAW_CUDA', '1')]
# ScaLAPACK
scalapack = True
# HDF5
hdf5 = False
gpaw/build/examples/davide/gpaw/setup/load-gpaw.sh deleted 100644 → 0
View file @ 38aa27ce
#!/bin/bash
module load cuda/9.2.88
module load cudnn/7.1.4--cuda--9.2.88
module load gnu/6.4.0
module load openmpi/3.1.0--gnu--6.4.0
source $CINECA_SCRATCH/lib/openblas-0.3.4-openmp/load.sh
source <PYTHONHOME>/load.sh
source $CINECA_SCRATCH/lib/scalapack-2.0.2/load.sh
export GPAW_SETUP_PATH=$CINECA_SCRATCH/lib/gpaw-setups-0.9.11271
export PATH=<BASE>/bin:$PATH
export PYTHONPATH=<BASE>/lib/python2.7/site-packages:$PYTHONPATH
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment