README.md 3.73 KB
Newer Older
Valeriu Codreanu's avatar
Valeriu Codreanu committed
1
2
3
NEMO 3.6, GYRE configuration
============================

Sagar Dolas's avatar
Sagar Dolas committed
4
created by juha.lento@csc.fi, 2016-05-16
Valeriu Codreanu's avatar
Valeriu Codreanu committed
5
6
7
8
9

Build and test documentation for NEMO 3.6 in GYRE
configuration. Example commands are tested in CSC's Cray XC40,
`sisu.csc.fi`.

Sagar Dolas's avatar
Sagar Dolas committed
10
updated by sagar.dolas@surfsara.nl in context of benchmarking efforts on Tier-0 supercomputing systems. 
Valeriu Codreanu's avatar
Valeriu Codreanu committed
11

Sagar Dolas's avatar
Sagar Dolas committed
12
Most of the following instructions should work as it is. The key part is to install XIOS because NEMO needs XIOS libraries to compile. The GYRE configuration is pretty easy to compile and run. Please also refer to following guide for reference. There is not one standard rule for compiling NEMO on all plateforms. Each plateform needs a different and unique approach to make NEMO work. The standard workflow is like this : 
Valeriu Codreanu's avatar
Valeriu Codreanu committed
13

Sagar Dolas's avatar
Sagar Dolas committed
14
15
16
17
1. Install XIOS. 
2. Install NEMO. 
3. Apply the patch as described here to measure step time : <https://software.intel.com/en-us/articles/building-and-running-nemo-on-xeon-processors>
4. Run NEMO in GYRE configuration.
Valeriu Codreanu's avatar
Valeriu Codreanu committed
18

Sagar Dolas's avatar
Sagar Dolas committed
19
20
Download NEMO and XIOS sources
------------------------------
Valeriu Codreanu's avatar
Valeriu Codreanu committed
21
22
23

### Check out NEMO sources

Sagar Dolas's avatar
Sagar Dolas committed
24
25
26
`
svn co -r6542 http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM
`
Valeriu Codreanu's avatar
Valeriu Codreanu committed
27
28
29

### Check out XIOS2 sources

Sagar Dolas's avatar
Sagar Dolas committed
30
<http://www.nemo-ocean.eu/Using-NEMO/User-Guides/Basics/XIOS-IO-server-installation-and-use>
Valeriu Codreanu's avatar
Valeriu Codreanu committed
31
32
33
34
35
36
37
38
39
40
41

```
svn co -r819 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk xios-2.0
```


Build XIOS
----------

### Build environment

Sagar Dolas's avatar
Sagar Dolas committed
42
Xios requires Netcdf4. Please load the appropriate HDF5 and NetCDF4 modules. You might have to change the path in the configuration file. 
Valeriu Codreanu's avatar
Valeriu Codreanu committed
43
44
45
46
47
48
49
50

```
module load cray-hdf5-parallel cray-netcdf-hdf5parallel
```


### Build command

Sagar Dolas's avatar
Sagar Dolas committed
51
<http://forge.ipsl.jussieu.fr/ioserver/wiki/documentation>
Valeriu Codreanu's avatar
Valeriu Codreanu committed
52
53
54

```
cd xios-2.0
Sagar Dolas's avatar
Sagar Dolas committed
55
./make_xios --arch XC30_Cray
Valeriu Codreanu's avatar
Valeriu Codreanu committed
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
```

Build NEMO 3.6 in GYRE configuration
------------------------------------

### Get a bash helper for editing configuration files

```
source <(curl -s https://raw.githubusercontent.com/jlento/nemo/master/fixfcm.bash)
```

...or if you have a buggy bash 3.2...

```
wget https://raw.githubusercontent.com/jlento/nemo/master/fixfcm.bash; source fixfcm.bash
```


### Edit (create) configuration files

```
cd ../NEMOGCM/CONFIG
fixfcm < ../ARCH/arch-XC40_METO.fcm > ../ARCH/arch-MY_CONFIG.fcm \
	NCDF_HOME="$NETCDF_DIR" \
	HDF5_HOME="$HDF5_DIR" \
	XIOS_HOME="$(readlink -f ../../xios-2.0)"
```


### Build

```
./makenemo -m MY_CONFIG -r GYRE_XIOS -n MY_GYRE add_key "key_nosignedzero"
```

Sagar Dolas's avatar
Sagar Dolas committed
91
Apply the patch as described here to measure step time : <https://software.intel.com/en-us/articles/building-and-running-nemo-on-xeon-processors>, otherwise it becomes extermely difficult to measure computational time.
Valeriu Codreanu's avatar
Valeriu Codreanu committed
92
93
94
95

Run first GYRE test
-------------------

Sagar Dolas's avatar
Sagar Dolas committed
96
97
Adjust the following configuration or settings as per your HPC site's preferences.

Valeriu Codreanu's avatar
Valeriu Codreanu committed
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
### Preapare input files

```
cd MY_GYRE/EXP00
sed -i '/using_server/s/false/true/' iodef.xml
sed -i '/&nameos/a ln_useCT = .false.' namelist_cfg
sed -i '/&namctl/a nn_bench = 1' namelist_cfg
```

### Run the experiment interactively

```
aprun -n 4 ../BLD/bin/nemo.exe : -n 2 ../../../../xios-2.0/bin/xios_server.exe
```


GYRE configuration with higher resolution
-----------------------------------------

### Modify configuration

Parameter `jp_cfg` controls the resolution.

```
rm -f time.step solver.stat output.namelist.dyn ocean.output  slurm-*  GYRE_* mesh_mask_00*
jp_cfg=4
sed -i -r \
    -e 's/^( *nn_itend *=).*/\1 21600/' \
    -e 's/^( *nn_stock *=).*/\1 21600/' \
    -e 's/^( *nn_write *=).*/\1 1000/' \
    -e 's/^( *jp_cfg *=).*/\1 '"$jp_cfg"'/' \
    -e 's/^( *jpidta *=).*/\1 '"$(( 30 * jp_cfg +2))"'/' \
    -e 's/^( *jpjdta *=).*/\1 '"$(( 20 * jp_cfg +2))"'/' \
    -e 's/^( *jpiglo *=).*/\1 '"$(( 30 * jp_cfg +2))"'/' \
    -e 's/^( *jpjglo *=).*/\1 '"$(( 20 * jp_cfg +2))"'/' \
    namelist_cfg

```