# PRACE 5IP - Task 6.2 - Service 6: ImageNet use cases with different frameworks.

## 1-IF you don't have the preprocessed data:

1. Place under Dataset/RawData/ the file ILSVRC2012_img_train.tar, recovered from the web (you can use as well the script developed by CINECA).

2. Place under Dataset/RawData/ the file ILSVRC2012_img_val.tar, recovered from the web (you can use as well the script developed by CINECA).

3. Place under Dataset/BoundingBoxes/ the file ILSVRC2012_bbox_train_v2.tar.gz, recovered from the web.

4. Run the prepro experiments (see below).

## 2-IF you have the preprocessed data available, the preprocessed files can be copied from the previous use case to their corresponding folders:

1. Tensorflow experiments: TFRecords in Dataset/PostProcessedTFRecords/

2. Caffe experiments: LMDB files in Dataset/PostProcessedLMDB/train and Dataset/PostProcessedLMDB/val

3. Keras experimentes: They were not included in the previous version, so preprocessing phase should be run at least once in order to run them.

## 3- Check set_environment.sh in order to choose a TMPDIR directory for the preprocessing steps (with plenty or no quota limitation)


## 4- Run one of the implemented tests by running the script:

```
master_script.sh <test> <num_gpus> [<bs>]
```
Being: 
test: < caffe | tf1.1 | tf1.8 | tf-horo | keras >
num_gpus: < prepo | 1 | 2 | 4 >
bs: batch size to be used: by default 32

The list of implemented tests within the script are listed below. Batch size can be set or ommited, in which case it will be left by default in 32:

### Caffe
```
master_script.sh caffe prepro
master_script.sh caffe 1 [<bs>]
master_script.sh caffe 2 [<bs>]
master_script.sh caffe 4 [<bs>]
```
Description: caffe framework running the prepro phase, or one epoch of VGG-19 training with 1, 2, or 4 GPUs.


### Tensorflow r1.1 legacy code
```
master_script.sh tf1.1 prepro
master_script.sh tf1.1 1 [<bs>]
master_script.sh tf1.1 2 [<bs>]
master_script.sh tf1.1 4 [<bs>]
```
Description: former code from the r1.1 branch running the prepro phase, or one epoch of ResNet50 training with 1, 2, or 4 GPUs.


### Tensorflow r1.8 code
```
master_script.sh tf1.8 prepro
master_script.sh tf1.8 1 [<bs>]
master_script.sh tf1.8 2 [<bs>]
master_script.sh tf1.8 4 [<bs>]
```
Description: newer code 1.8 compatible running the prepro phase, or one epoch of ResNet50 training with 1, 2, or 4 GPUs.

### Distrib TF r1.8 with horovod protocol
```
master_script.sh tf-horo prepro
master_script.sh tf-horo 8 [<bs>]
master_script.sh tf-horo 16 [<bs>]
master_script.sh tf-horo 32 [<bs>]
```
Description: same code in a distributed setting using MPI+Horovod protocol, one epoch of ResNet50 training for 8, 16, or 32 GPU.


### Distrib Keras + TF + Horovod.
```
master_script.sh keras prepro
master_script.sh keras 8 [<bs>]
master_script.sh keras 16 [<bs>]
master_script.sh keras 32 [<bs>]
```
Description: Keras-based code using TF as backend and Horovod, one epoch of ResNet50 training for 8, 16, or 32 GPU.

Example not performing well yet.


### Distrib TF r1.8 with gRPC protocol

EXAMPLE NOT WORKING YET.

