Newer
Older
Code_Saturne 4.2.2 is linked to PETSC developer's version, in order to benefit from
its GPU implementation. Note that the normal release of PETSC does not support GPU.
The version has been tested for K80s, and with the following settings:
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
To install Code_Saturne 4.2.2, 4 libraries are required, BLAS, LAPACK, SOWING and CUSP.
The tests have been carried out with lapack-3.6.1 for BLAS and LAPACK, sowing-1.1.23-p1
for SOWING and cusplibrary-0.5.1 for CUSP.
PETSC is first installed, and `PATH_TO_PETSC`, `PATH_TO_CUSP`, `PATH_TO_SOWING`, `PATH_TO_LAPACK`
have to be updated in `INSTALL_PETSC_GPU_sm37` under `petsc-petsc-a31f61e8abd0`
PETSC is configured for K80s, `./INSTALL_PETSC_GPU_sm37` is used from `petsc-petsc-a31f61e8abd0`
It is finally compiled and installed, by typing `make` and `make install`.
Before installing Code\_Saturne, adapt `PATH_TO_PETSC` in `InstallHPC.sh` under `SATURNE_4.2.2`
and type `./InstallHPC.sh`
The code should be installed and code_saturne be found under:
```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne
```
And should return:
```
Usage: ./code_saturne <topic>
Topics:
help
autovnv
bdiff
bdump
compile
config
create
gui
info
run
salome
submit
Options:
-h, --help show this help message and exit
```
### Test case - Cavity 13M
In `CAVITY_13M.tar.gz` are found the mesh+partitions and 2 sets of subroutines, one for CPU and the
second one for GPU, i.e.:
```
CAVITY_13M/PETSC_CPU/SRC/*
CAVITY_13M/PETSC_GPU/SRC/*
CAVITY_13M/MESH/mesh_input_13M
CAVITY_13M/MESH/partition_METIS_5.1.0/*
```
To prepare a run, it is required to set up a "study" with 2 directories, one for CPU and the other one for GPU
as, for instance:
```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_CAVITY_13M PETSC_CPU
cd NEW_CAVITY_13M
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --case PETSC_GPU
```
The mesh has to be copied from `CAVITY_13M/MESH/mesh_input_13M` into `NEW_CAVITY_13M/MESH/`.
And the same has to be done for `partition_METIS_5.1.0`.
The subroutines contained in `CAVITY_13M/PETSC_CPU/SRC` should be copied into `NEW_CAVITY_13M/PETSC_CPU/SRC` and
the subroutines contained in `CAVITY_13M/PETSC_GPU/SRC` should be copied into `NEW_CAVITY_13M/PETSC_GPU/SRC`.
In each DATA subdirectory of `NEW_CAVITY_13M/PETSC_CPU` and `NEW_CAVITY_13M/PETSC_GPU`, the path
to the mesh+partition has to be set as:
```
cd DATA
cp REFERENCE/cs_user_scripts.py .
edit cs_user_scripts.py:
* At line 138, change `None` to `../MESH/mesh_input_13M`
* At line 139, change `None` to `../MESH/partition_METIS_5.1.0`
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
At this stage, everything is set to run both simulations, one for the CPU and the other one for the GPU.
```
cd NEW_CAVITY_13M/PETSC_CPU
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```
Then submit the job.
```
cd NEW_CAVITY_13M/PETSC_GPU
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```
submit the job
## KNL Version
Code_Saturne 4.2.2 is installed for KNLs. It is also linked to PETSC, but the
default linear solvers are the native ones.
### Installation
The installation script is:
```
SATURNE_4.2.2/InstallHPC_with_PETSc.sh
```
The path to PETSC (official released version, and therefore assumed to be installed
on the machine) should be added to the aforementioned script.
After typing `./InstallHPC_with_PETSc.sh` the code should be installed and code_saturne be found under:
```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne
```
And should return:
```
Usage: ./code_saturne <topic>
Topics:
help
autovnv
bdiff
bdump
compile
config
create
gui
info
run
salome
submit
Options:
-h, --help show this help message and exit
```
### Running the code
Two cases are dealt with, `TGV_256_CS_OPENMP.tar.gz` to test the native solvers, and
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
`CAVITY_13M_FOR_KNLs_WITH_PETSC.tar.gz` to test Code_Saturne and PETSC on KNLs
#### First test case: TGV_256_CS_OPENMP
In `TGV_256_CS_OPENMP.tar.gz` are found the mesh and the set of subroutines for Code_Saturne on KNLs, i.e.:
```
TGV_256_CS_OPENMP/MESH/mesh_input_256by256by256
TGV_256_CS_OPENMP/ARCHER_KNL/SRC/*
```
To prepare a run, it is required to set up a "study" as, for instance:
```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_TGV_256_CS_OPENMP KNL
```
The mesh has to be copied from `TGV_256_CS_OPENMP/MESH/mesh_input_256by256by256` into `NEW_TGV_256_CS_OPENMP/MESH/`.
The subroutines contained in `TGV_256_CS_OPENMP/KNL/SRC` should be copied into `NEW_TGV_256_CS_OPENMP/KNL/SRC`.
In the `DATA` subdirectory of `NEW_TGV_256_CS_OPENMP/KNL` the path to the mesh has to be set as:
```
cd DATA
cp REFERENCE/cs_user_scripts.py .
```
Then edit `cs_user_scripts.py`:
At line 138, change `None` to `../MESH/mesh_input_256by256by256`
At this stage, everything is set to run the simulation:
```
cd TGV_256_CS_OPENMP/KNL
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```
And then submit the job.
#### Second test case: CAVITY_13M_FOR_KNLs_WITH_PETSC
In `CAVITY_13M.tar.gz` are found the mesh and the set of subroutines for PETSC and KNLs, i.e.:
```
CAVITY_13M/MESH/mesh_input
CAVITY_13M/KNL/SRC/*
```
To prepare a run, it is required to set up a "study" as, for instance:
```
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne create --study NEW_CAVITY_13M PETSC_KNL
```
The mesh has to be copied from `CAVITY_13M/MESH/mesh_input` into `NEW_CAVITY_13M/MESH/`.
The subroutines contained in `CAVITY_13M/KNL/SRC` should be copied into `NEW_CAVITY_13M/PETSC_KNL/SRC`.
In the `DATA` subdirectory of `NEW_CAVITY_13M/PETSC_KNL` the path to the mesh has to be set as:
```
cd DATA
cp REFERENCE/cs_user_scripts.py .
```
Then edit `cs_user_scripts.py`
At line 138, change `None` to `../MESH/mesh_input`
At this stage, everything is set to run the simulation:
```
cd NEW_CAVITY_13M/PETSC_KNL
PATH_TO_CODE_SATURNE/SATURNE_4.2.2/code_saturne-4.2.2/arch/Linux/bin/code_saturne run --initialize
cd RESU/YYYYMMDD-HHMM
```
Finaly submit the job