Skip to content
Commits on Source (6)
Expand all Show whitespace changes
Inline Side-by-side
gpaw/README.md
View file @ b57ca544
......@@ -2,7 +2,7 @@
## Summary version
0.1
0.9
## Purpose of the benchmark
......@@ -37,16 +37,23 @@ There is currently no active support for non-CUDA accelerator platforms.
For the UEABS benchmark version 2.2, the following versions of GPAW were tested:
* CPU-based:
* Version 1.5.2 as this one is the last of the 1.5 branch and since the GPU version
is derived from this version.
* Version 20.1.0, the most recent version during the development of the UEABS
2.2 benchmark suite.
* GPU-based: There is no official release or version number. The UEABS 2.2 benchmark
suite was tested using commit TODO of
[the cuda branch of the GitLab for CUDA development](https://gitlab.com/mlouhivu/gpaw/tree/cuda).
* Version 20.1.0, as this one is the version on which the most recent GPU commits
are based.
* Version 20.10.0, as it was the most recent version during the development of
the IEABS 2.2. benchmark suite.
* GPU-based: As there is no official release of the GPU version and as it is
at the moment of the release of the UEABS version 2.2 under heavy development
to also support AMD GPUs, there is no official support for the GPU version
([the cuda branch of the GitLab for CUDA development](https://gitlab.com/mlouhivu/gpaw/tree/cuda))
in UEABS version 2.2.
Versions 1.5.2 and 19.8.1 were also considered but are not compatible with the regular
input files provided here. Hence support for those versions of GPAW was dropped in
this version of the UEABS.
There are three benchmark cases, denotes S, M and L.
### Case S: Carbon nanotube
A ground state calculation for a carbon nanotube in vacuum. By default uses a
......@@ -57,21 +64,25 @@ on 100 cores of a modern x86 cluster.
Input file: [benchmark/1_S_carbon-nanotube/input.py](benchmark/1_S_carbon-nanotube/input.py)
This input file still works with version 1.5.2 and 19.8.1 of GPAW.
### Case M: Copper filament
A ground state calculation for a copper filament in vacuum. By default uses a
3x4x4 FCC lattice with 71 atoms (freely adjustable through the variables `x`,
`y` and `z` in the input file) and ScaLAPACK for
parallellisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
parallelisation. Expected to scale up to 100 nodes and/or 1000 MPI tasks.
Input file: [benchmark/2_M_copper-filament/input.py](benchmark/2_M_copper-filament/input.py)
The benchmark was tested using 1000 and 1024 cores. For some core configurations, one may
get error messages similar to ``gpaw.grid_descriptor.BadGridError: Grid ... to small
for ... cores``. If one really wants to run the benchmark for those number of cores,
one needs to adapt the values of `x`, `y` and `z` in `input.py`. However, this
changes the benchmark so results cannot be compared easily with benchmark runs for
different values of these variables.
This input file does not work with GPAW 1.5.2 and 19.8.1. It requires GPAW
20.1.0 or 20.10.0. Please try older versions of the UEABS if you want to use
these versions of GPAW.
The benchmark runs best when using full nodes. Expect a
performance drop on other configurations.
### Case L: Silicon cluster
......@@ -82,10 +93,16 @@ and/or 10000 MPI tasks.
Input file: [benchmark/3_L_silicon-cluster/input.py](benchmark/3_L_silicon-cluster/input.py)
This input file does not work with GPAW 1.5.2 and 19.8.1. It requires GPAW
20.1.0 or 20.10.0. Please try older versions of the UEABS if you want to use
these versions of GPAW.
## Mechanics of building the benchmark
Note that GPAW version numbering changed in 2019. Version 1.5.3 is the
Installing and running GPAW has changed a lot in the since the previous
versions of the UEABS. GPAW version numbering changed in 2019. Version 1.5.3 is the
last version with the old numbering. In 2019 the development team switched
to a version numbering scheme based on year, month and patchlevel, e.g.,
19.8.1 for the second version released in August 2019.
......@@ -94,16 +111,11 @@ Another change is in the Python packages used to install GPAW. Versions up to
and including 19.8.1 use the `distutils` package while versions 20.1.0 and later
are based on `setuptools`. This does affect the installation process.
GPAW for a while supports two different ways to run in parallel distributed memory mode:
* Using a wrapper executable `gpaw-python` that replaces the Python interpreter (it internally
links to the libpython library) and that provides the MPI functionality.
* Using the standard Python interpreter, including the MPI functionality in the
`_gpaw.so` shared library.
In the `distutils`-based versions, the wrapper script approach is the default behaviour,
while in the `setuptools`-based versions, the approach using the standard Python interpreter
is the preferred one in the manual. Even though the code in the `setuptools`-based
versions still includes the option to use the wrapper script approach, it does not
work in the tested version 20.1.0.
Running GPAW is no longer done via a wrapper executable `gpaw-python` that
replaces the Python interpreter (it internally links to the libpython library)
and that provides the MPI functionality. Since version 20.1.0, the standard Python
interpreter is used and the MPI functionality is included in the `_gpaw.so` shared library.
### Available instructions
......@@ -128,26 +140,23 @@ Hence GPAW has the following requirements:
* BLAS, LAPACK, BLACS and ScaLAPACK. ScaLAPACK is optional for GPAW, but mandatory
for the UEABS benchmarks. It is used by the medium and large cases and optional
for the small case.
* Python. GPAW 1.5.2 requires
Python 2.7 or 3.4-3.7, GPAW 19.8.1 requires 3.4-3.7, GPAW 20.1.0 Python 3.5-3.8
and GPAW 20.10.0 Python 3.6-3.9.
* Python. GPAW 20.1.0 requires Python 3.5-3.8 and GPAW 20.10.0 Python 3.6-3.9.
* Mandatory Python packages:
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [NumPY](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 20.1.0/20.10.0)
GPAW versions before 20.10.0 produce warnings when used with NumPy 1.19.x.
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 20.1.0/20.10.0)
* [FFTW](http://www.fftw.org) is highly recommended. As long as the optional libvdwxc
component is not used, the MKL FFTW wrappers can also be used. Recent versions of
GPAW also show good performance using just the NumPy-provided FFT routines provided
that NumPy has been built with a highly optimized FFT library.
* [LibXC](https://www.tddft.org/programs/libxc/) 2.X or newer for GPAW 1.5.2,
3.X or 4.X for GPAW 19.8.1, 20.1.0 and 20.10.0. LibXC is a library
of exchange-correlation functions for density-functional theory. None of the
versions currently mentions LibXC 5.X as officially supported.
* [LibXC](https://www.tddft.org/programs/libxc/) 3.X or 4.X for GPAW 20.1.0 and 20.10.0.
LibXC is a library of exchange-correlation functions for density-functional theory.
None of the versions currently mentions LibXC 5.X as officially supported.
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW
* Check the release notes of GPAW as the releases of ASE and GPAW should match.
E.g., during the development of the UEABS version 2.2 benchamark suite,
version 20.1.0 was the most up-to-date release of GPAW with 3.19.1 the matching ASE version
(though 3.18.0 should also work).
The benchmarks were tested using ASE 3.19.3 with GPAW 20.1.0 and ASE 3.20.1
with GPAW 20.1.0.
* ASE has some optional dependencies that are not needed for the benchmarking: Matplotlib (2.0.0 or newer),
tkinter (Tk interface, part of the Standard Python Library) and Flask.
* Optional components of GPAW that are not used by the UEABS benchmarks:
......@@ -194,17 +203,15 @@ git clone -b cuda https://gitlab.com/mlouhivu/gpaw.git
### Install
Crucial for the configuration of GPAW is a proper `customize.py` (GPAW 19.8.1 and
earlier) or `siteconfig.py` (GPAW 20.1.0 and later) file. The defaults used by GPAW
Crucial for the configuration of GPAW is a proper `siteconfig.py` file (GPAW 20.1.0 and later,
earlier versions used `customize.py`). The defaults used by GPAW
may not offer optimal performance and the automatic detection of the libraries also
fails on some systems.
The UEABS repository contains additional instructions:
* [general instructions](build/build-cpu.md)
* [GPGPUs](build/build-cuda.md) - To check
Example [build scripts](build/examples/) are also available for some PRACE and non-PRACE
systems.
Example [build scripts](build/examples/) are also available.
## Mechanics of Running the Benchmark
......@@ -221,26 +228,9 @@ right from this repository.
### Running the benchmarks
#### Using the `gpaw-python` wrapper script
This is the default approach for versions up to and including 19.8.1 of GPAW
These instructions are exclusively for GPAW 20.1.0 and later.
These versions of GPAW come with their own wrapper executable, `gpaw-python`,
to start a MPI-based GPAW run.
No special command line options or environment variables are needed to run the
benchmarks if your MPI process starter (`mpirun`, Slurm `srun`, ...) communicates
properly with the resource manager. E.g., on Slurm systems, use
```
srun gpaw-python input.py
```
#### Using the regular Python interpreter and parallel GPAW shared library
This is the default method for GPAW 20.1.0 (and likely later).
The wrapper executable `gpaw-python` is no longer available in the default parallel
build of GPAW. There are now two different ways to start GPAW.
There are two different ways to start GPAW.
One way is through `mpirun`, `srun` or an equivalent process starter and the
`gpaw python` command:
......@@ -263,13 +253,6 @@ That option however doesn't do the imports in the same way that the `gpaw` scrip
would do.
### Examples
Example [job scripts](scripts/) (`scripts/job-*.sh`) are provided for
different PRACE systems that may offer a helpful starting point.
*TODO: Update the examples as testing on other systems goes on.*
## Verification of Results
### Case S: Carbon nanotube
......
gpaw/build/build-CPU.md
View file @ b57ca544
......@@ -7,11 +7,11 @@ These instructions are in addition to the brief instructions in [README.md](../R
### Libraries and Python interpreter
GPAW needs (for the UEABS benchmarks)
* [Python](https://www.python.org/): GPAW 1.5.2 supports Python 2.7 and 3.4-3.7.
GPAW 19.8.1 needs Python 3.4-3.7 and GPAW 20.1.0 requires Python 3.5-3.8.
* [Python](https://www.python.org/): GPAW 20.1.0 requires Python 3.5-3.8, and
GPAW 20.10.0 and 21.1.0 require Python 3.6-3.9.
* [MPI library](https://www.mpi-forum.org/)
* [LibXC](https://www.tddft.org/programs/libxc/). GPAW 1.5.2 requires LibXC 1.5.2
or later. GPAW 19.8.1 and 20.1.0 need LibXC 3.x or 4.x.
* [LibXC](https://www.tddft.org/programs/libxc/). GPAW 20.1.0,
20.10.0 and 21.1.0 all need LibXC 3.x or 4.x.
* (Optimized) [BLAS](http://www.netlib.org/blas/) and
[LAPACK](http://www.netlib.org/lapack/) libraries.
There are both commercial and free and open source versions of these libraries.
......@@ -48,15 +48,15 @@ Optional components of GPAW that are not used by the UEABS benchmarks:
GPAW needs
* [wheel](https://pypi.org/project/wheel/) is needed in most (if not all) ways of
installing the packages from source.
* [NumPy](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [NumPy](https://pypi.org/project/numpy/) 1.9 or later (for GPAW 20.1.0/20.10.0/21.1.0)
* Installing NumPy from source will also require
[Cython](https://pypi.org/project/Cython/)
* GPAW 1.5.2 is not fully compatible with NumPy 1.19.x. Warnings about the use
* GPAW 20.1.0 is not fully compatible with NumPy 1.19.x or later. Warnings about the use
of deprecated constructs will be shown.
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 1.5.2/19.8.1/20.1.0/20.10.0)
* [SciPy](https://pypi.org/project/scipy/) 0.14 or later (for GPAW 20.1.0/20.10.0/21.1.0)
* [ASE, Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/), a Python package
from the same group that develops GPAW. Required versions are 3.17.0 or later for
GPAW 1.5.2 and 3.18.0 or later for GPAW 19.8.1 or 20.1.0.
from the same group that develops GPAW. The required versions is 3.18.0 or later for
GPAW 20.1.0, 20.10.0 and 21.1.0.
ASE has a couple of dependendencies
that are not needed for running the UEABS benchmarks. However, several Python
package install methods will trigger the installation of those packages, and
......@@ -119,11 +119,11 @@ GPAW needs
The table below give the combinations of major packages Python, NumPy, SciPy, ASE and
GPAW that were tested:
| Python | NumPy | SciPy | ASE | GPAW |
|:-------|:-------|:------|:-------|:--------|
| 3.7.9 | 1.18.5 | 1.4.1 | 3.17.0 | 1.5.2 |
| 3.7.9 | 1.18.5 | 1.4.1 | 3.18.2 | 19.8.1 |
| 3.8.6 | 1.18.5 | 1.4.1 | 3.19.3 | 20.1.0 |
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.1 | 1.19.5 | 1.5.4 |
| 21.1.0 | 3.21.1 | 3.9.1 | 1.19.5 | 1.5.4 |
## Installing all prerequisites
......@@ -213,43 +213,10 @@ of NumPy, SciPy and GPAW itself proves much more important.
## Configuring and installing GPAW
### GPAW 1.5.2
* GPAW 1.5.2 uses `distutils`. Customization of the installation process is possible
through the `customize.py` file.
* The FFT library: According to the documentation, the following strategy is used
* The compile process searches (in this order) for ``libmkl_rt.so``,
``libmkl_intel_lp64.so`` and ``libfftw3.so`. First one found will be
loaded.
* If none is found, the built-in FFT from NumPy will be used. This does not need
to be a problem if NumPy provides a properly optimized FFT library.
* The choice can also be overwritten using the GPAW_FFTWSO environment variable.
* With certain compilers, the GPAW test suite produced crashes in `xc/xc.py`. The
patch for GPAW 1.5.2 included in the [pathces](patches) subdirectory solved these
problems on the systems tested.
### GPAW 19.8.1
* GPAW 19.8.1 uses `distutils`. Customization of the installation process is possible
through a `customize.py` file.
* The selection process of the FFT library has changed from version 1.5.2. It is
now possible to specify the FFT library in `customize.py` or to simply select to
use the NumPy FFT routines.
### GPAW 20.1.0 and 20.10.0
* GPAW 20.1.0 uses `setuptools`. Customization of the installation process is possible
* GPAW 20.1.0 and later use `setuptools`. Customization of the installation process is possible
through the `siteconfig.py` file.
* The selection process of the FFT library is the same as in version 19.8.1, except
that the settings are now in `siteconfrig.py` rather than `customize.py`.
### All versions
* It is possible to specify the FFT library in `siteconfrig.py or to simply select to
use the NumPy FFT routines.
* GPAW also needs a number of so-called "Atomic PAW Setup" files. The latest files
can be found on the [GPAW website, Atomic PAW Setups page](https://wiki.fysik.dtu.dk/gpaw/setups/setups.html).
For the testing we used []`gpaw-setups-0.9.20000.tar.gz`](https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz)
......@@ -257,8 +224,7 @@ of NumPy, SciPy and GPAW itself proves much more important.
the file and set the environment variable GPAW_SETUP_PATH to point to that directory.
In the examples provided we use the `share/gpaw-setups` subdirectory of the install
directory for this purpose.
* Up to and including version 20.1.0, GPAW does comes with a test suite which can be
used after installation.
* GPAW 20.1.0 comes with a test suite which can be used after installation.
* Running the sequential tests:
gpaw test
......@@ -288,7 +254,7 @@ of NumPy, SciPy and GPAW itself proves much more important.
* Depending on the Python installation, some tests may fail with error messages that point
to a package in the Standard Python Library that is not present. Some of these errors have no
influence on the benchmarks as that part of the code is not triggered by the benchmark.
* The full test suite is missing in GPAW 20.10.0. There is a brief sequential test
* The full test suite is missing in GPAW 20.10.0 and later. There is a brief sequential test
that can be run with
gpaw test
......@@ -298,7 +264,7 @@ of NumPy, SciPy and GPAW itself proves much more important.
gpaw -P 4 test
* Multiple versions of GPAW likely contain a bug in `c/bmgs/fd.c` (around line 44
in GPAW 1.5.2). The code enforces vectorization on OpenMP 4 compilers by using
in GPAW 20.1.0). The code enforces vectorization on OpenMP 4 compilers by using
`#pragma omp simd`. However, it turns out that the data is not always correctly
aligned, so if the reaction of the compiler to `#pragma omp simd` is to fully vectorize
and use load/store instructions for aligned data, crashes may occur. It did happen
......@@ -312,6 +278,6 @@ of NumPy, SciPy and GPAW itself proves much more important.
wrappers in the 2020 compilers. Downgrading to the MKL libraries of the 2018
compilers or using the FFTW libraries solves the problem.
This has been observed not only in GPAW, but also in some other DFT packages.
* The GPAW test code in versions 1.5.2 till 20.1.0 detects that matplotlib is not installed
* The GPAW test code in version 20.1.0 detects if matplotlib is not installed
and will skip this test. We did however observe a failed test when Python could not find
the SQLite package as the Python standard library sqlite3 package is used.
gpaw/build/examples/CalcUA-leibniz-broadwell/README.md 0 → 100644
View file @ b57ca544
# Example build scripts for CalcUA-leibniz-broadwell
Cluster characteristics:
* CPU architecture: dual-socket 14-core Intel Xeon E5-2680v4 "broadwell" CPUs
* Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
* Package manager: Most software on the cluster is installed through [
EasyBuild](https://easybuilders.github.io/).
Unless otherwise mentioned below, the following combinations of versions of GPAW, ASE, Python, NumPy
and SciPy were used:
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.1 | 1.19.5 | 1.5.4 |
| 21.1.0 | 3.21.1 | 3.9.1 | 1.19.5 | 1.5.4 |
The following example build scripts do a complete build, including a build of Python
and a minimal set of dependencies that is needed to build a large enough part of the
Python standard library to build GPAW and run the benchmarks: zlib,
ncurses, readline, libffi, SQLite (used by one of the GPAW tests but likely
not needed for the benchmarks), FFTW, libxc, Python and the Python packages
NumPy, SciPy, ase and GPAW. The included libraries are not sufficient for a
full Python installation with all standard library packages or a full ASE/GPAW
installation with all optional packages but should be enough to run the benchmarks.
All software is build with the Intel 2020 compilers.
* `build_20.1.0_Python38_FFTW_icc`
* `build_20.10.0_Python39_FFTW_icc`
* `build_21.1.0_Python39_FFTW_icc`
The following build example build scripts install GPAW on top of the Intel Python
distribution. They rely as much as possible on modules already in the Intel Python
distribution or software that should be installed in the OS to run Intel Python.
The configuration accesses the FFT library available through
NumPy rather then linking directly to a FFT library.
The example script does use a module that provides up-to-date build
tools (buildtools/2020a) but on recent OSes it can be omitted.
Rather than loading the Intel and IntelPython3 modules one could as well
run the scripts provided by Intel to initialize the environment variables
for the compilers and Python distribution, and provided the necessary build
tools are also available in the system OS, this build could be done without
relying on any EasyBuild or Spack-generated modules.
* `build_20.1.0_IntelPython3_icc.sh`
* `build_20.10.0_IntelPython3_icc.sh`
* `build_21.1.0_IntelPython3_icc.sh`
gpaw/build/examples/CalcUA-vaughan-rome/build_1.5.2_IntelPython3_icc.sh → gpaw/build/examples/CalcUA-leibniz-broadwell/build_20.1.0_IntelPython3_icc.sh
View file @ b57ca544
#!/bin/bash
#
# Installation script for GPAW 1.5.2:
# Installation script for GPAW 20.1.0:
# * Using the existing IntelPython3 module on the system which has an optimized
# NumPy and SciPy included.
# * Using the matching version of ase, 3.17.0
# * Using the matching version of ase, 3.19.3
# * Compiling with the Intel compilers
#
# The FFT library is discovered at runtime. With the settings used in this script
......@@ -12,40 +12,80 @@
# (link below).
#
# The original installation instructions for GPAW can be found at
# https://gitlab.com/gpaw/gpaw/-/blob/1.5.2/doc/install.rst
# https://gitlab.com/gpaw/gpaw/-/blob/20.1.0/doc/install.rst
#
packageID='1.5.2-IntelPython3-icc'
packageID='20.1.0-IntelPython3-icc'
echo -e "\n### Building GPAW-UEABS/$packageID from $0\n\n"
install_root=$VSC_SCRATCH/UEABS
systemID=CalcUA-vaughan-rome
systemID=CalcUA-leibniz-broadwell
UEABS_version='2.2'
#
# NOTE: For easy maintanance we load many settings from the UEABS module.
# Since we want to release the module again and only depend on system modules for
# this build, we copy the settings to other variables and unload the module
# again. It is of course possible to those variables directly by hand without
# first creating a module file.
#
download_dir=$install_root/Downloads
install_dir=$install_root/$systemID/Packages/GPAW-manual/$packageID
modules_dir=$install_root/$systemID/Modules/GPAW-manual
build_dir="/dev/shm/$USER/GPAW-manual/$packageID"
patch_dir=$VSC_DATA/Projects/PRACE/GPAW-experiments/UEABS/build/patches
module purge
module use $install_root/$systemID/Modules
module load UEABS/$UEABS_version
# Directory in which downloaded files will be stored.
download_dir=$UEABS_DOWNLOADS
# Directory in which the software package will be installed.
install_dir=$UEABS_PACKAGES/GPAW-UEABS/$packageID
# Subdirectory containing the module file.
modules_dir=$UEABS_MODULES/GPAW-UEABS
# Temporary directory for the build.
build_dir="/dev/shm/$USER/GPAW-UEABS/$packageID"
# Subdirectory in the UEABS repo containing the patch file that is needed.
patch_dir=$UEABS_REPOSITORY/gpaw/build/patches
#
# Software versions
#
IntelPython3_version='2020.02'
libxc_version='4.3.4'
ase_version='3.17.0'
GPAW_version='1.5.2'
ase_version='3.19.3'
GPAW_version='20.1.0'
GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
# Compiler settings
compiler_module='intel/2020a'
opt_level='-O2'
proc_opt_flags='-march=core-avx2 -mtune=core-avx2'
fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
parallel=32
py_maj_min='3.7'
################################################################################
#
# Prepare the system
#
# We do this through modules, but it could be done by running scripts from the
# Intel compiler suite as well.
#
#
# Load modules
# Load modules:
# - Intel compiler module
# - Intel Python
# - buildtools is a module with a number of build tools that are more up-to-date
# then those on the system. It is mostly used for consistent builds across
# systems.
#
module purge
module load calcua/2020a
module load intel/2020a
module load IntelPython3/2020a
MODULEPATH=/apps/antwerpen/modules/centos7/calcua-admin-broadwell/
module load calcua/admin
module load $compiler_module
module load buildtools/2020a
module load IntelPython3/$IntelPython3_version
#
# Create the directories and make sure they are clean if that matters
......@@ -63,7 +103,6 @@ module load buildtools/2020a
/usr/bin/mkdir -p $build_dir
################################################################################
#
# Download components
......@@ -78,17 +117,14 @@ libxc_file="libxc-$libxc_version.tar.bz2"
libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
# We do not download ase in this script. As it is pure python and doesn't need
# customization, we will install it using pip right away.
## https://files.pythonhosted.org/packages/d9/08/35969da23b641d3dfca46ba7559f651fcfdca81dbbc00b9058c934e75769/ase-3.17.0.tar.gz
#ase_file="ase-$ase_version.tar.gz"
#ase_url="https://files.pythonhosted.org/packages/d9/08/35969da23b641d3dfca46ba7559f651fcfdca81dbbc00b9058c934e75769"
#[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# Downloading ase so that we can gather all sources for reproducibility
ase_file="ase-$ase_version.tar.gz"
ase_url="https://pypi.python.org/packages/source/a/ase"
[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# GPAW needs customization, so we need to download and unpack the sources.
# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
GPAW_file="gpaw-$GPAW_version.tar.gz"
#GPAW_url="https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9"
GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
......@@ -117,16 +153,15 @@ cd libxc-$libxc_version
autoreconf -i
export CC=icc
export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -fPIC"
export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC"
#export CFLAGS="-O0 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -fPIC"
export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
./configure --prefix="$install_dir" \
--disable-static --enable-shared --disable-fortran
# Build
make -j 16
make -j $parallel
# Install
make -j 16 install
make install
# Add bin, lib and include to the PATH variables
PATH=$install_dir/bin:$PATH
......@@ -134,6 +169,37 @@ LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
CPATH=$install_dir/include:$CPATH
# Clean-up
unset CC
unset CFLAGS
################################################################################
#
# Prepare for installing Python packages
#
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages:$PYTHONPATH"
################################################################################
#
# Optional: Install pytest and its dependencies to test NumPy and SciPy with
# import numpy
# numpy.test()
# import scipy
# scipy.text()
# We don't care about version numbers here as it is not important for the
# reproducibility of the benchmarks.
#
echo -e "\nInstalling pytest...\n"
cd $build_dir
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir pytest
################################################################################
#
......@@ -142,10 +208,12 @@ CPATH=$install_dir/include:$CPATH
echo -e "\nInstalling ase...\n"
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages"
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir --no-deps $download_dir/$ase_file
pip install --prefix=$install_dir --no-deps ase==$ase_version
# Brief test
cd $build_dir
python$py_maj_min -c "import ase"
################################################################################
......@@ -172,47 +240,38 @@ cd $build_dir
# Uncompress
tar -xf $download_dir/$GPAW_file
# Apply patches
patch -p0 <$patch_dir/gpaw-1.5.2.patch
cd gpaw-$GPAW_version
# Make the customize.py script
mv customize.py customize.py.orig
cat >customize.py <<EOF
print( 'GPAW build INFO: Starting execution of the customization script' )
print( 'GPAW build INFO: Variables at the start of the customization script' )
print( 'GPAW build INFO: libraries = ', libraries )
print( 'GPAW build INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW build INFO: library_dirs = ', library_dirs )
print( 'GPAW build INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW build INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW build INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW build INFO: include_dirs = ', include_dirs )
print( 'GPAW build INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW build INFO: compiler = ', compiler )
print( 'GPAW build INFO: mpicompiler = ', mpicompiler )
print( 'GPAW build INFO: mpilinker = ', mpilinker )
print( 'GPAW build INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW build INFO: extra_link_args = ', extra_link_args )
print( 'GPAW build INFO: define_macros = ', define_macros )
print( 'GPAW build INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW build INFO: undef_macros = ', undef_macros )
print( 'GPAW build INFO: scalapack = ', scalapack )
print( 'GPAW build INFO: libvdwxc = ', libvdwxc )
print( 'GPAW build INFO: elpa = ', elpa )
# Reset the lists of libraries as often the wrong BLAS library is picked up.
libraries = []
mpi_libraries = []
# LibXC. Re-add the library (removed by resetting libraries).
# There is no need to add the library directory as we do set library_dirs to the
# content of LIBRARY_PATH further down.
# There should be no need to add the include directory as it is in CPATH which is
# set when we install gpaw.
#include_dirs.append('$install_dir/include')
libraries.append('xc')
cat >siteconfig.py <<EOF
print( 'GPAW EasyBuild INFO: Starting execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the start of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
# LibXC
include_dirs.append('$install_dir/include')
#libraries.append('xc')
# libvdwxc
libvdwxc = False
......@@ -220,17 +279,17 @@ libvdwxc = False
# ELPA
elpa = False
# Use NumPy FFTW
fftw = False
# ScaLAPACK
scalapack = True
mpi_libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
mpi_define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
mpi_define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
# Add EasyBuild LAPACK/BLAS libs
# This should also enable MKL FFTW according to the documentation of GPAW 1.5.2
libraries += ['mkl_intel_lp64', 'mkl_sequential', 'mkl_core']
# MKL BLAS
libraries += ['mkl_sequential','mkl_core', 'mkl_rt', ]
# Add other EasyBuild library directories.
# Add other EasyBuild library directoryes.
library_dirs = os.environ['LIBRARY_PATH'].split(':')
# Set the compilers
......@@ -238,44 +297,49 @@ compiler = os.environ['CC']
mpicompiler = os.environ['MPICC']
mpilinker = os.environ['MPICC']
# We need extra_compile_args to have the right compiler options when re-compiling
# files for gpaw-python. It does imply double compiler options for the other
# compiles though.
extra_compile_args = os.environ['CFLAGS'].split(' ')
print( 'GPAW build INFO: Variables at the end of the customization script' )
print( 'GPAW build INFO: libraries = ', libraries )
print( 'GPAW build INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW build INFO: library_dirs = ', library_dirs )
print( 'GPAW build INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW build INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW build INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW build INFO: include_dirs = ', include_dirs )
print( 'GPAW build INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW build INFO: compiler = ', compiler )
print( 'GPAW build INFO: mpicompiler = ', mpicompiler )
print( 'GPAW build INFO: mpilinker = ', mpilinker )
print( 'GPAW build INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW build INFO: extra_link_args = ', extra_link_args )
print( 'GPAW build INFO: define_macros = ', define_macros )
print( 'GPAW build INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW build INFO: undef_macros = ', undef_macros )
print( 'GPAW build INFO: scalapack = ', scalapack )
print( 'GPAW build INFO: libvdwxc = ', libvdwxc )
print( 'GPAW build INFO: elpa = ', elpa )
print( 'GPAW build INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the end of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
EOF
# Now install gpaw
export CC=icc
export MPICC=mpiicc
export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -qno-openmp-simd"
export CFLAGS="-O2 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -qno-openmp-simd"
#export CFLAGS="-O0 -march=core-avx2 -mtune=core-avx2 -ftz -fp-speculation=safe -fp-model source -qno-openmp-simd"
python setup.py build -j 32
export CFLAGS="-std=c99 $opt_level $proc_opt_flags $fp_opt_flags -qno-openmp-simd"
python$py_maj_min setup.py build -j $parallel
# Install GPAW
python setup.py install --prefix="$install_dir"
pip install --prefix=$install_dir --no-deps --ignore-installed --no-build-isolation .
# Brief test
cd $build_dir
python$py_maj_min -c "import gpaw"
# Clean-up
unset CC
unset MPICC
unset CFLAGS
################################################################################
......@@ -283,14 +347,17 @@ python setup.py install --prefix="$install_dir"
# Finish the install
#
echo -e "\nCleaning up and making the LUA-module GPAW-manual/$packageID...\n"
echo -e "\nCleaning up and making the LUA-module GPAW-UEABS/$packageID...\n"
# Go to a different directory before cleaning up the build directory
cd $modules_dir
#/bin/rm -rf $build_dir
/bin/rm -rf $build_dir
# Create a module file
python_version=$(python -V | cut -d ' ' -f 2)
numpy_version=$(python3 -c "import numpy ; print(numpy.__version__)")
scipy_version=$(python3 -c "import scipy ; print(scipy.__version__)")
cat >$packageID.lua <<EOF
help([==[
......@@ -300,14 +367,23 @@ Description
GPAW $GPAW_version for the UEABS benchmark.
Configuration:
* IntelPython3/2020a module, also for NumPy and SciPy
* Parallel GPAW $GPAW_version with ase $ase_version
* FFT library selected at runtime. The default with the path
as set through this module should be MKL but it can be changed
by setting GPAW_FFTWSO as indicated in the install instructions at
https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/install.rst
* On top of the system IntelPython3 $IntelPython3_version module, which provides
Python $python_version, NumPy $numpy_version and SciPy $scipy_version.
* NumPy is used for FFT, so the performance of the code
will depend a lot on a proper configuration of NumPy for FFT.
* libxc and GPAW compiled with the Intel compilers
Detailed configuration:
* Compiler module: $compiler_module (a module file written
at UAntwerp, not generated via EasyBuild)
* Modules from the system
* Intel Python3 $IntelPython3_version (using a module file written
at UAntwerp)
* libxc $libxc_version
* Python packages: ase-$ase_version, gpaw-$GPAW_version
* GPAW setups $GPAWsetups_version
More information
================
......@@ -316,31 +392,25 @@ More information
- GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
- Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
- ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
Included extensions
===================
ase-$ase_version, gpaw-$GPAW_version
]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version: UEABS benchmark configuration.]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version, Intel Python3 $IntelPython3_version with numpy $numpy_version and scipy $scipy_version: UEABS benchmark configuration.]==])
conflict("GPAW")
conflict("GPAW-manual")
family("GPAW")
if not ( isloaded("calcua/2020a") ) then
load("calcua/2020a")
end
prepend_path('MODULEPATH','/apps/antwerpen/modules/centos8/calcua-admin-rome')
if not ( isloaded("intel/2020a") ) then
load("intel/2020a")
if not ( isloaded("calcua/admin") ) then
load("calcua/admin")
end
if not ( isloaded("IntelPython3/2020a") ) then
load("IntelPython3/2020a")
if not ( isloaded("$compiler_module") ) then
load("$compiler_module")
end
if not ( isloaded("IntelPython3/$IntelPython3_version") ) then
load("IntelPython3/$IntelPython3_version")
end
prepend_path("PATH", "$install_dir/bin")
prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
......
gpaw/build/examples/CalcUA-leibniz-broadwell/build_20.10.0_IntelPython3_icc.sh 0 → 100755
View file @ b57ca544
#!/bin/bash
#
# Installation script for GPAW 20.10.0:
# * Using the existing IntelPython3 module on the system which has an optimized
# NumPy and SciPy included.
# * Using the matching version of ase, 3.20.1
# * Compiling with the Intel compilers
#
# The FFT library is discovered at runtime. With the settings used in this script
# this should be MKL FFT, but it is possible to change this at runtime to either
# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
# (link below).
#
# The original installation instructions for GPAW can be found at
# https://gitlab.com/gpaw/gpaw/-/blob/20.10.0/doc/install.rst
#
packageID='20.10.0-IntelPython3-icc'
echo -e "\n### Building GPAW-UEABS/$packageID from $0\n\n"
install_root=$VSC_SCRATCH/UEABS
systemID=CalcUA-leibniz-broadwell
UEABS_version='2.2'
#
# NOTE: For easy maintanance we load many settings from the UEABS module.
# Since we want to release the module again and only depend on system modules for
# this build, we copy the settings to other variables and unload the module
# again. It is of course possible to those variables directly by hand without
# first creating a module file.
#
module purge
module use $install_root/$systemID/Modules
module load UEABS/$UEABS_version
# Directory in which downloaded files will be stored.
download_dir=$UEABS_DOWNLOADS
# Directory in which the software package will be installed.
install_dir=$UEABS_PACKAGES/GPAW-UEABS/$packageID
# Subdirectory containing the module file.
modules_dir=$UEABS_MODULES/GPAW-UEABS
# Temporary directory for the build.
build_dir="/dev/shm/$USER/GPAW-UEABS/$packageID"
# Subdirectory in the UEABS repo containing the patch file that is needed.
patch_dir=$UEABS_REPOSITORY/gpaw/build/patches
#
# Software versions
#
IntelPython3_version='2020.02'
libxc_version='4.3.4'
ase_version='3.20.1'
GPAW_version='20.10.0'
GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
# Compiler settings
compiler_module='intel/2020a'
opt_level='-O2'
proc_opt_flags='-march=core-avx2 -mtune=core-avx2'
fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
parallel=32
py_maj_min='3.7'
################################################################################
#
# Prepare the system
#
# We do this through modules, but it could be done by running scripts from the
# Intel compiler suite as well.
#
#
# Load modules:
# - Intel compiler module
# - Intel Python
# - buildtools is a module with a number of build tools that are more up-to-date
# then those on the system. It is mostly used for consistent builds across
# systems.
#
module purge
MODULEPATH=/apps/antwerpen/modules/centos7/calcua-admin-broadwell/
module load calcua/admin
module load $compiler_module
module load buildtools/2020a
module load IntelPython3/$IntelPython3_version
#
# Create the directories and make sure they are clean if that matters
#
/usr/bin/mkdir -p $download_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/rm -rf $install_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/mkdir -p $modules_dir
/usr/bin/mkdir -p $build_dir
/usr/bin/rm -rf $build_dir
/usr/bin/mkdir -p $build_dir
################################################################################
#
# Download components
#
echo -e "\nDownloading files...\n"
cd $download_dir
# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
libxc_file="libxc-$libxc_version.tar.bz2"
libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
# Downloading ase so that we can gather all sources for reproducibility
ase_file="ase-$ase_version.tar.gz"
ase_url="https://pypi.python.org/packages/source/a/ase"
[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# GPAW needs customization, so we need to download and unpack the sources.
# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
GPAW_file="gpaw-$GPAW_version.tar.gz"
GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
# Download GPAW-setup, a number of setup files for GPAW.
# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
################################################################################
#
# Install libxc
#
echo -e "\nInstalling libxc...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$libxc_file
cd libxc-$libxc_version
# Configure
autoreconf -i
export CC=icc
export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
./configure --prefix="$install_dir" \
--disable-static --enable-shared --disable-fortran
# Build
make -j $parallel
# Install
make install
# Add bin, lib and include to the PATH variables
PATH=$install_dir/bin:$PATH
LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
CPATH=$install_dir/include:$CPATH
# Clean-up
unset CC
unset CFLAGS
################################################################################
#
# Prepare for installing Python packages
#
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages:$PYTHONPATH"
################################################################################
#
# Optional: Install pytest and its dependencies to test NumPy and SciPy with
# import numpy
# numpy.test()
# import scipy
# scipy.text()
# We don't care about version numbers here as it is not important for the
# reproducibility of the benchmarks.
#
echo -e "\nInstalling pytest...\n"
cd $build_dir
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir pytest
################################################################################
#
# Install ase
#
echo -e "\nInstalling ase...\n"
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir --no-deps $download_dir/$ase_file
# Brief test
cd $build_dir
python$py_maj_min -c "import ase"
################################################################################
#
# Install GPAW-setups
#
echo -e "\nInstalling gpaw-setups...\n"
mkdir -p $install_dir/share/gpaw-setups
cd $install_dir/share/gpaw-setups
tar -xf $download_dir/$GPAWsetups_file --strip-components=1
################################################################################
#
# Install GPAW
#
echo -e "\nInstalling GPAW...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$GPAW_file
cd gpaw-$GPAW_version
# Make the customize.py script
cat >siteconfig.py <<EOF
print( 'GPAW EasyBuild INFO: Starting execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the start of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
# LibXC
include_dirs.append('$install_dir/include')
#libraries.append('xc')
# libvdwxc
libvdwxc = False
# ELPA
elpa = False
# Use NumPy FFTW
fftw = False
# ScaLAPACK
scalapack = True
libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
# MKL BLAS
libraries += ['mkl_sequential','mkl_core', 'mkl_rt', ]
# Add other EasyBuild library directoryes.
library_dirs = os.environ['LIBRARY_PATH'].split(':')
# Set the compilers
compiler = os.environ['CC']
mpicompiler = os.environ['MPICC']
mpilinker = os.environ['MPICC']
print( 'GPAW EasyBuild INFO: Variables at the end of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
EOF
# Now install gpaw
export CC=icc
export MPICC=mpiicc
export CFLAGS="-std=c99 $opt_level $proc_opt_flags $fp_opt_flags -qno-openmp-simd"
python$py_maj_min setup.py build -j $parallel
# Install GPAW
pip install --prefix=$install_dir --no-deps --ignore-installed --no-build-isolation .
# Brief test
cd $build_dir
python$py_maj_min -c "import gpaw"
# Clean-up
unset CC
unset MPICC
unset CFLAGS
################################################################################
#
# Finish the install
#
echo -e "\nCleaning up and making the LUA-module GPAW-UEABS/$packageID...\n"
# Go to a different directory before cleaning up the build directory
cd $modules_dir
/bin/rm -rf $build_dir
# Create a module file
python_version=$(python -V | cut -d ' ' -f 2)
numpy_version=$(python3 -c "import numpy ; print(numpy.__version__)")
scipy_version=$(python3 -c "import scipy ; print(scipy.__version__)")
cat >$packageID.lua <<EOF
help([==[
Description
===========
GPAW $GPAW_version for the UEABS benchmark.
Configuration:
* Parallel GPAW $GPAW_version with ase $ase_version
* On top of the system IntelPython3 $IntelPython3_version module, which provides
Python $python_version, NumPy $numpy_version and SciPy $scipy_version.
* NumPy is used for FFT, so the performance of the code
will depend a lot on a proper configuration of NumPy for FFT.
* libxc and GPAW compiled with the Intel compilers
Detailed configuration:
* Compiler module: $compiler_module (a module file written
at UAntwerp, not generated via EasyBuild)
* Modules from the system
* Intel Python3 $IntelPython3_version (using a module file written
at UAntwerp)
* libxc $libxc_version
* Python packages: ase-$ase_version, gpaw-$GPAW_version
* GPAW setups $GPAWsetups_version
More information
================
- Homepage: http://wiki.fysik.dtu.dk/gpaw
- Documentation:
- GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
- Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
- ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version, Intel Python3 $IntelPython3_version with numpy $numpy_version and scipy $scipy_version: UEABS benchmark configuration.]==])
family("GPAW")
prepend_path('MODULEPATH','/apps/antwerpen/modules/centos8/calcua-admin-rome')
if not ( isloaded("calcua/admin") ) then
load("calcua/admin")
end
if not ( isloaded("$compiler_module") ) then
load("$compiler_module")
end
if not ( isloaded("IntelPython3/$IntelPython3_version") ) then
load("IntelPython3/$IntelPython3_version")
end
prepend_path("PATH", "$install_dir/bin")
prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
prepend_path("LIBRARY_PATH", "$install_dir/lib")
prepend_path("PYTHONPATH", "$install_dir/lib/python$py_maj_min/site-packages")
setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
EOF
gpaw/build/examples/CalcUA-leibniz-broadwell/build_21.1.0_IntelPython3_icc.sh 0 → 100755
View file @ b57ca544
#!/bin/bash
#
# Installation script for GPAW 21.1.0:
# * Using the existing IntelPython3 module on the system which has an optimized
# NumPy and SciPy included.
# * Using the matching version of ase, 3.21.1
# * Compiling with the Intel compilers
#
# The FFT library is discovered at runtime. With the settings used in this script
# this should be MKL FFT, but it is possible to change this at runtime to either
# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
# (link below).
#
# The original installation instructions for GPAW can be found at
# https://gitlab.com/gpaw/gpaw/-/blob/21.1.0/doc/install.rst
#
packageID='21.1.0-IntelPython3-icc'
echo -e "\n### Building GPAW-UEABS/$packageID from $0\n\n"
install_root=$VSC_SCRATCH/UEABS
systemID=CalcUA-leibniz-broadwell
UEABS_version='2.2'
#
# NOTE: For easy maintanance we load many settings from the UEABS module.
# Since we want to release the module again and only depend on system modules for
# this build, we copy the settings to other variables and unload the module
# again. It is of course possible to those variables directly by hand without
# first creating a module file.
#
module purge
module use $install_root/$systemID/Modules
module load UEABS/$UEABS_version
# Directory in which downloaded files will be stored.
download_dir=$UEABS_DOWNLOADS
# Directory in which the software package will be installed.
install_dir=$UEABS_PACKAGES/GPAW-UEABS/$packageID
# Subdirectory containing the module file.
modules_dir=$UEABS_MODULES/GPAW-UEABS
# Temporary directory for the build.
build_dir="/dev/shm/$USER/GPAW-UEABS/$packageID"
# Subdirectory in the UEABS repo containing the patch file that is needed.
patch_dir=$UEABS_REPOSITORY/gpaw/build/patches
#
# Software versions
#
IntelPython3_version='2020.02'
libxc_version='4.3.4'
ase_version='3.21.1'
GPAW_version='21.1.0'
GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
# Compiler settings
compiler_module='intel/2020a'
opt_level='-O2'
proc_opt_flags='-march=core-avx2 -mtune=core-avx2'
fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
parallel=32
py_maj_min='3.7'
################################################################################
#
# Prepare the system
#
# We do this through modules, but it could be done by running scripts from the
# Intel compiler suite as well.
#
#
# Load modules:
# - Intel compiler module
# - Intel Python
# - buildtools is a module with a number of build tools that are more up-to-date
# then those on the system. It is mostly used for consistent builds across
# systems.
#
module purge
MODULEPATH=/apps/antwerpen/modules/centos7/calcua-admin-broadwell/
module load calcua/admin
module load $compiler_module
module load buildtools/2020a
module load IntelPython3/$IntelPython3_version
#
# Create the directories and make sure they are clean if that matters
#
/usr/bin/mkdir -p $download_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/rm -rf $install_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/mkdir -p $modules_dir
/usr/bin/mkdir -p $build_dir
/usr/bin/rm -rf $build_dir
/usr/bin/mkdir -p $build_dir
################################################################################
#
# Download components
#
echo -e "\nDownloading files...\n"
cd $download_dir
# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
libxc_file="libxc-$libxc_version.tar.bz2"
libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
# Downloading ase so that we can gather all sources for reproducibility
ase_file="ase-$ase_version.tar.gz"
ase_url="https://pypi.python.org/packages/source/a/ase"
[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# GPAW needs customization, so we need to download and unpack the sources.
# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
GPAW_file="gpaw-$GPAW_version.tar.gz"
GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
# Download GPAW-setup, a number of setup files for GPAW.
# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
################################################################################
#
# Install libxc
#
echo -e "\nInstalling libxc...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$libxc_file
cd libxc-$libxc_version
# Configure
autoreconf -i
export CC=icc
export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
./configure --prefix="$install_dir" \
--disable-static --enable-shared --disable-fortran
# Build
make -j $parallel
# Install
make install
# Add bin, lib and include to the PATH variables
PATH=$install_dir/bin:$PATH
LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
CPATH=$install_dir/include:$CPATH
# Clean-up
unset CC
unset CFLAGS
################################################################################
#
# Prepare for installing Python packages
#
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages:$PYTHONPATH"
################################################################################
#
# Optional: Install pytest and its dependencies to test NumPy and SciPy with
# import numpy
# numpy.test()
# import scipy
# scipy.text()
# We don't care about version numbers here as it is not important for the
# reproducibility of the benchmarks.
#
echo -e "\nInstalling pytest...\n"
cd $build_dir
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir pytest
################################################################################
#
# Install ase
#
echo -e "\nInstalling ase...\n"
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir --no-deps $download_dir/$ase_file
# Brief test
cd $build_dir
python$py_maj_min -c "import ase"
################################################################################
#
# Install GPAW-setups
#
echo -e "\nInstalling gpaw-setups...\n"
mkdir -p $install_dir/share/gpaw-setups
cd $install_dir/share/gpaw-setups
tar -xf $download_dir/$GPAWsetups_file --strip-components=1
################################################################################
#
# Install GPAW
#
echo -e "\nInstalling GPAW...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$GPAW_file
cd gpaw-$GPAW_version
# Make the customize.py script
cat >siteconfig.py <<EOF
print( 'GPAW EasyBuild INFO: Starting execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the start of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
# LibXC
include_dirs.append('$install_dir/include')
#libraries.append('xc')
# libvdwxc
libvdwxc = False
# ELPA
elpa = False
# Use NumPy FFTW
fftw = False
# ScaLAPACK
scalapack = True
libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
# MKL BLAS
libraries += ['mkl_sequential','mkl_core', 'mkl_rt', ]
# Add other EasyBuild library directoryes.
library_dirs = os.environ['LIBRARY_PATH'].split(':')
# Set the compilers
compiler = os.environ['CC']
mpicompiler = os.environ['MPICC']
mpilinker = os.environ['MPICC']
print( 'GPAW EasyBuild INFO: Variables at the end of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
EOF
# Now install gpaw
export CC=icc
export MPICC=mpiicc
export CFLAGS="-std=c99 $opt_level $proc_opt_flags $fp_opt_flags -qno-openmp-simd"
python$py_maj_min setup.py build -j $parallel
# Install GPAW
pip install --prefix=$install_dir --no-deps --ignore-installed --no-build-isolation .
# Brief test
cd $build_dir
python$py_maj_min -c "import gpaw"
# Clean-up
unset CC
unset MPICC
unset CFLAGS
################################################################################
#
# Finish the install
#
echo -e "\nCleaning up and making the LUA-module GPAW-UEABS/$packageID...\n"
# Go to a different directory before cleaning up the build directory
cd $modules_dir
/bin/rm -rf $build_dir
# Create a module file
python_version=$(python -V | cut -d ' ' -f 2)
numpy_version=$(python3 -c "import numpy ; print(numpy.__version__)")
scipy_version=$(python3 -c "import scipy ; print(scipy.__version__)")
cat >$packageID.lua <<EOF
help([==[
Description
===========
GPAW $GPAW_version for the UEABS benchmark.
Configuration:
* Parallel GPAW $GPAW_version with ase $ase_version
* On top of the system IntelPython3 $IntelPython3_version module, which provides
Python $python_version, NumPy $numpy_version and SciPy $scipy_version.
* NumPy is used for FFT, so the performance of the code
will depend a lot on a proper configuration of NumPy for FFT.
* libxc and GPAW compiled with the Intel compilers
Detailed configuration:
* Compiler module: $compiler_module (a module file written
at UAntwerp, not generated via EasyBuild)
* Modules from the system
* Intel Python3 $IntelPython3_version (using a module file written
at UAntwerp)
* libxc $libxc_version
* Python packages: ase-$ase_version, gpaw-$GPAW_version
* GPAW setups $GPAWsetups_version
More information
================
- Homepage: http://wiki.fysik.dtu.dk/gpaw
- Documentation:
- GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
- Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
- ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version, Intel Python3 $IntelPython3_version with numpy $numpy_version and scipy $scipy_version: UEABS benchmark configuration.]==])
family("GPAW")
prepend_path('MODULEPATH','/apps/antwerpen/modules/centos8/calcua-admin-rome')
if not ( isloaded("calcua/admin") ) then
load("calcua/admin")
end
if not ( isloaded("$compiler_module") ) then
load("$compiler_module")
end
if not ( isloaded("IntelPython3/$IntelPython3_version") ) then
load("IntelPython3/$IntelPython3_version")
end
prepend_path("PATH", "$install_dir/bin")
prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
prepend_path("LIBRARY_PATH", "$install_dir/lib")
prepend_path("PYTHONPATH", "$install_dir/lib/python$py_maj_min/site-packages")
setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
EOF
gpaw/build/examples/CalcUA-vaughan-rome/README.md 0 → 100644
View file @ b57ca544
# Example build scripts for CalcUA-vaughan-rome
Cluster characteristics:
* CPU architecture: dual-socket 32-core AMD EPYC 7452 "Rome" CPUs
* Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
* Package manager: Most software on the cluster is installed through [
EasyBuild](https://easybuilders.github.io/).
Unless otherwise mentioned below, the following combinations of versions of GPAW, ASE, Python, NumPy
and SciPy were used:
| GPAW | ASE | Python | NumPy | SciPy |
|:--------|:--------|:-------|:-------|:------|
| 20.1.0 | 3.19.3 | 3.8.7 | 1.18.5 | 1.5.4 |
| 20.10.0 | 3.20.1 | 3.9.1 | 1.19.5 | 1.5.4 |
| 21.1.0 | 3.21.1 | 3.9.1 | 1.19.5 | 1.5.4 |
The following example build scripts do a complete build, including a build of Python
and a minimal set of dependencies that is needed to build a large enough part of the
Python standard library to build GPAW and run the benchmarks: zlib,
ncurses, readline, libffi, SQLite (used by one of the GPAW tests but likely
not needed for the benchmarks), FFTW, libxc, Python and the Python packages
NumPy, SciPy, ase and GPAW. The included libraries are not sufficient for a
full Python installation with all standard library packages or a full ASE/GPAW
installation with all optional packages but should be enough to run the benchmarks.
All software is build with the Intel 2020 compilers.
* `build_20.1.0_Python38_FFTW_icc`
* `build_20.10.0_Python39_FFTW_icc`
* `build_21.1.0_Python39_FFTW_icc`
The following build example build scripts install GPAW on top of the Intel Python
distribution. They rely as much as possible on modules already in the Intel Python
distribution or software that should be installed in the OS to run Intel Python.
The configuration accesses the FFT library available through
NumPy rather then linking directly to a FFT library.
The example script does use a module that provides up-to-date build
tools (buildtools/2020a) but on recent OSes it can be omitted.
Rather than loading the Intel and IntelPython3 modules one could as well
run the scripts provided by Intel to initialize the environment variables
for the compilers and Python distribution, and provided the necessary build
tools are also available in the system OS, this build could be done without
relying on any EasyBuild or Spack-generated modules.
* `build_20.1.0_IntelPython3_icc.sh`
* `build_20.10.0_IntelPython3_icc.sh`
* `build_21.1.0_IntelPython3_icc.sh`
The following example build scripts depend on other modules (Python and FFTW) that
were installed on the cluster through EasyBuild beyond the compiler modules and a
module providing some up-to-date basic build tools:
* `build_20.1.0_Python39icc_icc.sh`
* `build_20.10.0_Python39icc_icc.sh`
* `build_21.1.0_Python39icc_icc.sh`
gpaw/build/examples/CalcUA-vaughan-rome/build_20.1.0_IntelPython3_icc.sh 0 → 100755
View file @ b57ca544
#!/bin/bash
#
# Installation script for GPAW 20.1.0:
# * Using the existing IntelPython3 module on the system which has an optimized
# NumPy and SciPy included.
# * Using the matching version of ase, 3.19.3
# * Compiling with the Intel compilers
#
# The FFT library is discovered at runtime. With the settings used in this script
# this should be MKL FFT, but it is possible to change this at runtime to either
# MKL, FFTW or the built-in NumPy FFT routines, see the installation instructions
# (link below).
#
# The original installation instructions for GPAW can be found at
# https://gitlab.com/gpaw/gpaw/-/blob/20.1.0/doc/install.rst
#
packageID='20.1.0-IntelPython3-icc'
echo -e "\n### Building GPAW-UEABS/$packageID from $0\n\n"
install_root=$VSC_SCRATCH/UEABS
systemID=CalcUA-vaughan-rome
UEABS_version='2.2'
#
# NOTE: For easy maintanance we load many settings from the UEABS module.
# Since we want to release the module again and only depend on system modules for
# this build, we copy the settings to other variables and unload the module
# again. It is of course possible to those variables directly by hand without
# first creating a module file.
#
module purge
module use $install_root/$systemID/Modules
module load UEABS/$UEABS_version
# Directory in which downloaded files will be stored.
download_dir=$UEABS_DOWNLOADS
# Directory in which the software package will be installed.
install_dir=$UEABS_PACKAGES/GPAW-UEABS/$packageID
# Subdirectory containing the module file.
modules_dir=$UEABS_MODULES/GPAW-UEABS
# Temporary directory for the build.
build_dir="/dev/shm/$USER/GPAW-UEABS/$packageID"
# Subdirectory in the UEABS repo containing the patch file that is needed.
patch_dir=$UEABS_REPOSITORY/gpaw/build/patches
#
# Software versions
#
IntelPython3_version='2020.00'
libxc_version='4.3.4'
ase_version='3.19.3'
GPAW_version='20.1.0'
GPAWsetups_version='0.9.20000' # Check version on https://wiki.fysik.dtu.dk/gpaw/setups/setups.html
# Compiler settings
compiler_module='intel/2020a.00'
opt_level='-O2'
proc_opt_flags='-march=core-avx2 -mtune=core-avx2'
fp_opt_flags='-ftz -fp-speculation=safe -fp-model source'
parallel=32
py_maj_min='3.7'
################################################################################
#
# Prepare the system
#
# We do this through modules, but it could be done by running scripts from the
# Intel compiler suite as well.
#
#
# Load modules:
# - Intel compiler module
# - Intel Python
# - buildtools is a module with a number of build tools that are more up-to-date
# then those on the system. It is mostly used for consistent builds across
# systems.
#
module purge
MODULEPATH=/apps/antwerpen/modules/centos8/calcua-admin-rome/
module load calcua/admin
module load buildtools/2020a
module load IntelPython3/$IntelPython3_version
module load $compiler_module
#
# Create the directories and make sure they are clean if that matters
#
/usr/bin/mkdir -p $download_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/rm -rf $install_dir
/usr/bin/mkdir -p $install_dir
/usr/bin/mkdir -p $modules_dir
/usr/bin/mkdir -p $build_dir
/usr/bin/rm -rf $build_dir
/usr/bin/mkdir -p $build_dir
################################################################################
#
# Download components
#
echo -e "\nDownloading files...\n"
cd $download_dir
# https://gitlab.com/libxc/libxc/-/archive/4.3.4/libxc-4.3.4.tar.bz2
libxc_file="libxc-$libxc_version.tar.bz2"
libxc_url="https://gitlab.com/libxc/libxc/-/archive/$libxc_version"
[[ -f $libxc_file ]] || wget "$libxc_url/$libxc_file"
# Downloading ase so that we can gather all sources for reproducibility
ase_file="ase-$ase_version.tar.gz"
ase_url="https://pypi.python.org/packages/source/a/ase"
[[ -f $ase_file ]] || wget "$ase_url/$ase_file"
# GPAW needs customization, so we need to download and unpack the sources.
# https://files.pythonhosted.org/packages/49/a1/cf54c399f5489cfdda1e8da02cae8bfb4b39d7cb7a895ce86608fcd0e1c9/gpaw-1.5.2.tar.gz
GPAW_file="gpaw-$GPAW_version.tar.gz"
GPAW_url="https://pypi.python.org/packages/source/g/gpaw"
[[ -f $GPAW_file ]] || wget "$GPAW_url/$GPAW_file"
# Download GPAW-setup, a number of setup files for GPAW.
# https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-0.9.20000.tar.gz
GPAWsetups_file="gpaw-setups-$GPAWsetups_version.tar.gz"
GPAWsetups_url="https://wiki.fysik.dtu.dk/gpaw-files"
[[ -f $GPAWsetups_file ]] || wget "$GPAWsetups_url/$GPAWsetups_file"
################################################################################
#
# Install libxc
#
echo -e "\nInstalling libxc...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$libxc_file
cd libxc-$libxc_version
# Configure
autoreconf -i
export CC=icc
export CFLAGS="$opt_level $proc_opt_flags $fp_opt_flags -fPIC"
./configure --prefix="$install_dir" \
--disable-static --enable-shared --disable-fortran
# Build
make -j $parallel
# Install
make install
# Add bin, lib and include to the PATH variables
PATH=$install_dir/bin:$PATH
LIBRARY_PATH=$install_dir/lib:$LIBRARY_PATH
LD_LIBRARY_PATH=$install_dir/lib:$LD_LIBRARY_PATH
CPATH=$install_dir/include:$CPATH
# Clean-up
unset CC
unset CFLAGS
################################################################################
#
# Prepare for installing Python packages
#
/usr/bin/mkdir -p "$install_dir/lib/python$py_maj_min/site-packages"
PYTHONPATH="$install_dir/lib/python$py_maj_min/site-packages:$PYTHONPATH"
################################################################################
#
# Optional: Install pytest and its dependencies to test NumPy and SciPy with
# import numpy
# numpy.test()
# import scipy
# scipy.text()
# We don't care about version numbers here as it is not important for the
# reproducibility of the benchmarks.
#
echo -e "\nInstalling pytest...\n"
cd $build_dir
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir pytest
################################################################################
#
# Install ase
#
echo -e "\nInstalling ase...\n"
# IntelPython3 does not define pip3 or pip3.7.
pip install --prefix=$install_dir --no-deps $download_dir/$ase_file
# Brief test
cd $build_dir
python$py_maj_min -c "import ase"
################################################################################
#
# Install GPAW-setups
#
echo -e "\nInstalling gpaw-setups...\n"
mkdir -p $install_dir/share/gpaw-setups
cd $install_dir/share/gpaw-setups
tar -xf $download_dir/$GPAWsetups_file --strip-components=1
################################################################################
#
# Install GPAW
#
echo -e "\nInstalling GPAW...\n"
cd $build_dir
# Uncompress
tar -xf $download_dir/$GPAW_file
cd gpaw-$GPAW_version
# Make the customize.py script
cat >siteconfig.py <<EOF
print( 'GPAW EasyBuild INFO: Starting execution of the customization script' )
print( 'GPAW EasyBuild INFO: Variables at the start of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
# LibXC
include_dirs.append('$install_dir/include')
#libraries.append('xc')
# libvdwxc
libvdwxc = False
# ELPA
elpa = False
# Use NumPy FFTW
fftw = False
# ScaLAPACK
scalapack = True
libraries += ['mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64']
# MKL BLAS
libraries += ['mkl_sequential','mkl_core', 'mkl_rt', ]
# Add other EasyBuild library directoryes.
library_dirs = os.environ['LIBRARY_PATH'].split(':')
# Set the compilers
compiler = os.environ['CC']
mpicompiler = os.environ['MPICC']
mpilinker = os.environ['MPICC']
print( 'GPAW EasyBuild INFO: Variables at the end of the customization script' )
print( 'GPAW EasyBuild INFO: libraries = ', libraries )
print( 'GPAW EasyBuild INFO: mpi_libaries = ', mpi_libraries )
print( 'GPAW EasyBuild INFO: library_dirs = ', library_dirs )
print( 'GPAW EasyBuild INFO: mpi_libary_dirs = ', mpi_library_dirs )
print( 'GPAW EasyBuild INFO: runtime_library_dirs = ', runtime_library_dirs )
print( 'GPAW EasyBuild INFO: mpi_runtime_libary_dirs = ', mpi_runtime_library_dirs )
print( 'GPAW EasyBuild INFO: include_dirs = ', include_dirs )
print( 'GPAW EasyBuild INFO: mpi_include_dirs = ', mpi_include_dirs )
print( 'GPAW EasyBuild INFO: compiler = ', compiler )
print( 'GPAW EasyBuild INFO: mpicompiler = ', mpicompiler )
print( 'GPAW EasyBuild INFO: mpilinker = ', mpilinker )
print( 'GPAW EasyBuild INFO: extra_compile_args = ', extra_compile_args )
print( 'GPAW EasyBuild INFO: extra_link_args = ', extra_link_args )
print( 'GPAW EasyBuild INFO: define_macros = ', define_macros )
print( 'GPAW EasyBuild INFO: mpi_define_macros = ', mpi_define_macros )
print( 'GPAW EasyBuild INFO: undef_macros = ', undef_macros )
print( 'GPAW EasyBuild INFO: fftw = ', fftw )
print( 'GPAW EasyBuild INFO: scalapack = ', scalapack )
print( 'GPAW EasyBuild INFO: libvdwxc = ', libvdwxc )
print( 'GPAW EasyBuild INFO: elpa = ', elpa )
print( 'GPAW EasyBuild INFO: Ending execution of the customization script' )
print( 'GPAW EasyBuild INFO: noblas = ', noblas )
print( 'GPAW EasyBuild INFO: parallel_python_interpreter = ', parallel_python_interpreter )
EOF
# Now install gpaw
export CC=icc
export MPICC=mpiicc
export CFLAGS="-std=c99 $opt_level $proc_opt_flags $fp_opt_flags -qno-openmp-simd"
python$py_maj_min setup.py build -j $parallel
# Install GPAW
pip install --prefix=$install_dir --no-deps --ignore-installed --no-build-isolation .
# Brief test
cd $build_dir
python$py_maj_min -c "import gpaw"
# Clean-up
unset CC
unset MPICC
unset CFLAGS
################################################################################
#
# Finish the install
#
echo -e "\nCleaning up and making the LUA-module GPAW-UEABS/$packageID...\n"
# Go to a different directory before cleaning up the build directory
cd $modules_dir
/bin/rm -rf $build_dir
# Create a module file
python_version=$(python -V | cut -d ' ' -f 2)
numpy_version=$(python3 -c "import numpy ; print(numpy.__version__)")
scipy_version=$(python3 -c "import scipy ; print(scipy.__version__)")
cat >$packageID.lua <<EOF
help([==[
Description
===========
GPAW $GPAW_version for the UEABS benchmark.
Configuration:
* Parallel GPAW $GPAW_version with ase $ase_version
* On top of the system IntelPython3 $IntelPython3_version module, which provides
Python $python_version, NumPy $numpy_version and SciPy $scipy_version.
* NumPy is used for FFT, so the performance of the code
will depend a lot on a proper configuration of NumPy for FFT.
* libxc and GPAW compiled with the Intel compilers
Detailed configuration:
* Compiler module: $compiler_module (a module file written
at UAntwerp, not generated via EasyBuild)
* Modules from the system
* Intel Python3 $IntelPython3_version (using a module file written
at UAntwerp)
* libxc $libxc_version
* Python packages: ase-$ase_version, gpaw-$GPAW_version
* GPAW setups $GPAWsetups_version
More information
================
- Homepage: http://wiki.fysik.dtu.dk/gpaw
- Documentation:
- GPAW web-based documentation: https://wiki.fysik.dtu.dk/gpaw/
- Version information at https://gitlab.com/gpaw/gpaw/-/blob/$GPAW_version/doc/
- ASE web-based documentation: https://wiki.fysik.dtu.dk/ase/
]==])
whatis([==[Description: GPAW $GPAW_version with ase $ase_version, Intel Python3 $IntelPython3_version with numpy $numpy_version and scipy $scipy_version: UEABS benchmark configuration.]==])
family("GPAW")
prepend_path('MODULEPATH','/apps/antwerpen/modules/centos8/calcua-admin-rome')
if not ( isloaded("calcua/admin") ) then
load("calcua/admin")
end
--
-- The order of loading is a deliberate choice. It takes some libraries from
-- the compiler directories rather than from the Python directories and has
-- solved some unexplained issues we had with SLURM.
--
if not ( isloaded("IntelPython3/$IntelPython3_version") ) then
load("IntelPython3/$IntelPython3_version")
end
if not ( isloaded("$compiler_module") ) then
load("$compiler_module")
end
prepend_path("PATH", "$install_dir/bin")
prepend_path("LD_LIBRARY_PATH", "$install_dir/lib")
prepend_path("LIBRARY_PATH", "$install_dir/lib")
prepend_path("PYTHONPATH", "$install_dir/lib/python$py_maj_min/site-packages")
setenv("GPAW_SETUP_PATH", "$install_dir/share/gpaw-setups")
EOF
gpaw/build/examples/README.md 0 → 100644
View file @ b57ca544
# Example build scripts
## CalcUA-vaughan-rome
Vaughan is a cluster installed at the University of Antwerpen
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 144 nodes with dual 32-core AMD EPYC 7452 "Rome" CPUs and 256GB of RAM memory
per node. Nodes are interconnected through an InfiniBand HDR100 network. The system
does not contain accelerators. The cluster uses CentOS 8 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
[Go to the directory](CalcUA-vaughan-rome)
## CalcUA-leibniz/broadwell
Leibniz is a cluster intalled at the University of Antwerpen
as one of the VSC (Vlaams Supercomputer Centrum) Tier-2 clusters.
The cluster has 153 regular CPU nodes with dual Intel Xeon E5-2680v4
"broadwell" CPUs with 128 or 256 GB of RAM memory per node.
Nodes are interconnected through an Infiniband EDR network.
The system has two nodes with dual NVIDIA P100 GPUs and a node
with dual NEC Aurora TSUBASA first generation vector boards.
During the testing period the cluster used CentOS 7 as the
operating system.
Most software on the cluster is installed through [EasyBuild](https://easybuilders.github.io/).
Environment modules are provided through [Lmod](https://lmod.readthedocs.io/en/latest/).
## davide @ Cineca
Build instructions for GPU and CPU on the DAVIDE OpenPower GPU cluster @ Cineca.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
## juwels @ Jülich Supercomputing Centre
Build instructions for the CPU nodes of JUWELS.
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.
## piz-daint @ CSCS
Build instructions for the Piz Daint (GPU cluster).
These instructions haven't been tested recently. They are still for a version
based on distutils rather then setuptools that was dropped towards the end of
the development of this version of the benchmark suite.