Push notifications in your browser are not yet configured.
You are not logged in, you may not see all content and functionalities. If you have an account, please login .
Problem with CFL-criterion and NaN values
Dear colleges,
I’ve faced serious problem in work with CCLM model: very often my simulations lead to appearance of NAN -values in the output with message “!!!!*** WARNING ***!!! CFL -criterion for horizontal advection is violated”. I’m trying to solve this problem by reducing time step dt, but now, when I work with version 5.0_clm2, and try different options from PHYCTL and DYNCTL , I noticed that no I need much lower value of dt that I used before with version 5.0 and standard parameters from CCLM -training.
For example, I launched model with 0.045 deg. horizontal resolution and dt=30, and model has failed ( NAN in output since some moment), but in my previous experience I’ve used model with 0.025 deg. resolution and same value of dt=30 and it worked OK.
So, my question is: what other model perameters could affect appearance of this problem? What of them are most important?
Thanks in advance!
The namelist tool includes an information on the dependency of namelist parameters on dt. It may still not be complete, though:
nrdtau, crltau, nincrad, hincrad
However I guess it is very likely that the problem lies in the namelist settings itself.
Be aware that CCLM5.0 is still a pre-released version and there is a warning on the download page:
This pre-released version contains COSMO -model source code that has been evaluated for the forecast mode but not yet for climate simulations!
This also concerns the namelist settings.
Presently, the evaluation tests for a new released version are underway. The results are planned to be presented on the CLM -Community General Assembly end of September.
Dear Burkhardt, thank you very much for your hints. I’m working together with Mikhail and have faced to the same problems. I woud like to remark, that we have used the same model version and model parameters in winter and spring, and we didn’t have these problems during experiments runs. Technical support of our supercomputer ‘Lomonosov’ gave us a prompt about solving this problem. They says it may be associated with the accessible RAM capacity. Maybe, you have any expierence of RAM consumption optimization? I mean somethings about binaries’ compilation options, or running command options, or maybe any namelists parameters?
Additionally, we have a question about controlling the NaN values in output? Does any namelists parameters exist that controls output values and, for example, terminates the model run (not logs the warnings only)?
Thanks a lot for any recommendations!
I am not an expert in memory issues in CCLM . In one of the next version there will be the possibility to run the model with 32bit real variables instead of 64bit if my information is right. This may be an option then, but has to be carefully tested beforehand.
If you provide information on the compiler you use and attach your
Fopts
file maybe someone who uses the same compiler can comment on it.Regarding NaN values in the output: Most compilers have an option that causes the program to stop in case of NaN.
Dear Burkhardt, thank you for recommendations. I’m attaching Fopts file. There are a lot of commented lines because of we have chosen the final configuration between many variants. We use the following parameters and libraries: fortran compiler – mpif90, intel-13.1.0/intel-15.0.90; mpi compiler – openmpi-1.8.4-icc.
Thanks to anyone for suggestions about compiler options or parameters.
Regarding the NaN issue: Did you check the option
-fpe0
?No, I didn’t. Could you explain, what is this option? Where should I use it – in Fopts compiler, or somewhere else?
-fpe0
is an Intel compiler option. It can be used in Fopts in addition to the other compiler flags.Dear colleagues, I have faced this problem again. The long model run (1 year) starts well, but after a few days (or even a few hours) it gets NaN values in YU* files (attached) and in model point output (M_Yuzhno-Sakhalinsk). It doesn’t lead to model crash, but all model variables in output have got NaN (see slurm-1151743). I have done different variants of compilation ( CKEYS in Fopts):
CKEYS = -c -fpp -fno-alias -unroll0 -heap-arrays 1000 – it gives NaN
CKEYS = -c -fpp -fno-alias -unroll0 -heap-arrays 1000 -g -O0 -traceback – it gives NaN
CKEYS = -c -fpp -fno-alias -unroll0 -heap-arrays 1000 -fpe0 -g -O0 -traceback – it crashes according to (slurm-1151468)
CKEYS = -c -fpp -fno-alias -unroll0 -heap-arrays 1000 -fpe0 – it crashes according to (slurm-1151420)
Could you give any solvings to this problem? I’m attaching the script file too (cclm_Sakhalin_0.12.sh).
Thank you.