TensorFlow (https://www.tensorflow.org) is a popular open-source library for symbolic math and linear algebra, with particular optimization for neural-networks-based machine learning workflow. Maintained by Google, it is widely used for research and production in both the academia and the industry.
TensorFlow supports a wide variety of hardware platforms (CPUs, GPUs, TPUs), and can be scaled up to utilize multiple compute devices on a single or multiple compute nodes. The main objective of this benchmark is to profile the scaling behavior of TensorFlow on different hardware, and thereby provide a reference baseline of its performance for different sizes of applications.
TensorFlow supports a wide variety of hardware platforms (CPUs, GPUs, TPUs) and can be scaled up to utilize multiple computing devices on a single or multiple compute nodes. The main objective of this benchmark is to profile the scaling behavior of TensorFlow on different hardware, and thereby provide a reference baseline of its performance for different sizes of applications.
DeepGalaxy
===
...
...
@@ -11,9 +11,9 @@ There are many open-source datasets available for benchmarking TensorFlow, such
This should clone the full benchmark code to a local directory called `DeepGalaxy`. Enter this directory with `cd DeepGalaxy`.
In doing so, the latest version of the `DeepGalaxy` benchmark suite will be downloaded. Note that the latest version is not necessarily the most stable version, and there is no guarantee of backward compatability with older TensorFlow versions.
**Step 2**: Download the training dataset.
In the `DeepGalaxy` directory, download the training dataset. Depending on the benchmark size, there are three datasets available:
...
...
@@ -44,28 +44,46 @@ In the `DeepGalaxy` directory, download the training dataset. Depending on the b
where `output_bw_512.hdf5` is the training dataset downloaded in the previous step. Please change the file name if necessary. One could also change the other parameters, such as `--epochs`, `--batch-size`, and `--arch` according to the size of the benchmark. For example, the `EfficientNetB0` deep neural network is for small HPC systems, `EfficientNetB4` is for medium-size ones, and `EfficientNetB7` is for large systems. Also, if there are a lot of memory, increasing the `--batch-size` could improve the throughput. If the `--batch-size` parameter is too large, an out-of-memory error could occur.
where `output_bw_512.hdf5` is the training dataset downloaded in the previous step. Please change the file name if necessary. One could also change the other parameters, such as `--epochs`, `--batch-size`, and `--arch` according to the size of the benchmark. For example, the `EfficientNetB0` deep neural network is for small HPC systems, `EfficientNetB4` is for medium-size ones, and `EfficientNetB7` is for large systems. Also, shoudl the system memory permits, increasing the `--batch-size` could improve the throughput. If the `--batch-size` parameter is too large, an out-of-memory error could occur.
It is wise to save the output of the `mpirun` command to a text file, for example, `DeepGalaxy.np_4.out`.
The benchmark data of the training are written to the file `train_log.txt`.
**Step 4**: Repeat Step 3 with different `np`.
All the desired `np` settings are completed, we should have a bunch of output files on the local directory. For example, `DeepGalaxy.np_4.out`, `DeepGalaxy.np_8.out`, and so on. We could then extract the throughput using the following command:
By the time when all the desired `np` settings are completed, we should have all the throughput data written in the `train_log.txt` file. The content of the file looks like this:
The throughput can be read from the timing here, such as `173ms/sample`. Usually, this number is a bit larger in the first epoch, because `TensorFlow` needs to do some initialization in the first epoch. So we could pikc up the number from the 3rd or even 5th epoch when it is stablized.
Extract this number for different `np`, and see how this number changes a function of `np`. In a system with perfect (i.e., linear) scaling, this number should be constant. But in reality, this number should increase due to the communication overhead. Therefore, the growth of this number as a function of `np` tell us something about the scaling efficiency of the underlying system.
This output contains several useful information for use to derive the scaling efficiency of the HPC system:
-`num_workers`: the number of (MPI) workers. This is essentially equal to the `-np` parameter in the `mpirun` command. Do not confuse this with the number of (CPU) cores used, because one worker may make use of multiple cores. If GPUs are used, one worker is typically associated with one GPU card.
- images/sec (per worker): this is the throughput per worker
- images/sec (total): this is the total throughput of the system
Due to the initialization effect, the throughputs of the first two epochs are lower, so please read the throughput data from the third epoch onwards.
With the data of total throughput, we could calculate the scaling efficiency. In an ideal system, the total throughput scales linearly as a function of `num_workers`, and hence the scaling efficiency is 1. In practice, the scaling efficiency drops with more workers due to the communication overhead. The better connectivity the HPC system is, the better scaling efficiency.
