Problem with writing restart file – in #9: CCLM

in #9: CCLM

<p> Dear colleagues, I have a problem with the nesting simulation (cclm-to-cclm) at the second cclm stage ( <span class="caps"> COSMO </span> - <span class="caps"> CLM </span> 5 version). The model has been run on 4 months, from 05/2010 to 08/2010 for Moscow region. But the model was interrupted on 240 hours step. 240 hours is the nhour_restart value, therefore, I have suggested, the problem is associated with restart parameters. Moreover, the restart file wasn’t written, and I can’t resume my simulation from this step. Could you suggest, please, where the problem is. Maybe, it associated with many Namelist parameters. I have attached the log-file (slurm-721469.out), launching script (cclm5_nest.sh). Furthermore, the <span class="caps"> OUTPUT </span> file wasn’t written. <br/> Thank you very much for any suggestions! </p>

  @vladimirplatonov in #e7dd65c

<p> Dear colleagues, I have a problem with the nesting simulation (cclm-to-cclm) at the second cclm stage ( <span class="caps"> COSMO </span> - <span class="caps"> CLM </span> 5 version). The model has been run on 4 months, from 05/2010 to 08/2010 for Moscow region. But the model was interrupted on 240 hours step. 240 hours is the nhour_restart value, therefore, I have suggested, the problem is associated with restart parameters. Moreover, the restart file wasn’t written, and I can’t resume my simulation from this step. Could you suggest, please, where the problem is. Maybe, it associated with many Namelist parameters. I have attached the log-file (slurm-721469.out), launching script (cclm5_nest.sh). Furthermore, the <span class="caps"> OUTPUT </span> file wasn’t written. <br/> Thank you very much for any suggestions! </p>

Problem with writing restart file

VP wrote

Dear colleagues, I have a problem with the nesting simulation (cclm-to-cclm) at the second cclm stage ( COSMO - CLM 5 version). The model has been run on 4 months, from 05/2010 to 08/2010 for Moscow region. But the model was interrupted on 240 hours step. 240 hours is the nhour_restart value, therefore, I have suggested, the problem is associated with restart parameters. Moreover, the restart file wasn’t written, and I can’t resume my simulation from this step. Could you suggest, please, where the problem is. Maybe, it associated with many Namelist parameters. I have attached the log-file (slurm-721469.out), launching script (cclm5_nest.sh). Furthermore, the OUTPUT file wasn’t written.
Thank you very much for any suggestions!

View in channel
<p> Please check, if the file <code> 7780OUT_DIR/restarts/lrfd2010051100o </code> already exists. <br/> If so, either delete the file or better: <br/> In <code> io_utilities.f90 </code> in <code> open_file </code> around about line 770 change ‘ <span class="caps"> NEW </span> ’ to ‘ <span class="caps"> UNKNOWN </span> ’ for the file opening <br/> <pre><br/> CASE (‘bina’)</pre> </p> IF (my_id 0) THEN IF (ymode(1:1) ‘r’) <span class="caps"> THEN </span> <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> OLD </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> READ </span> ’, <span class="caps"> IOSTAT </span> =ierror) <span class="caps"> ELSEIF </span> ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) <span class="caps"> THEN </span> ! there is no append for restart-files. They are NEW ! <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> NEW </span> ’, &amp; <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> UNKNOWN </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> WRITE </span> ’, <span class="caps"> IOSTAT </span> =ierror) <p> <br/> and compile a new binary. <br/> This generally changed in the climate version, but not yet done for the “fresh” release of the version cosmo5.0 </p>

  @burkhardtrockel in #9cbe8d9

<p> Please check, if the file <code> 7780OUT_DIR/restarts/lrfd2010051100o </code> already exists. <br/> If so, either delete the file or better: <br/> In <code> io_utilities.f90 </code> in <code> open_file </code> around about line 770 change ‘ <span class="caps"> NEW </span> ’ to ‘ <span class="caps"> UNKNOWN </span> ’ for the file opening <br/> <pre><br/> CASE (‘bina’)</pre> </p> IF (my_id 0) THEN IF (ymode(1:1) ‘r’) <span class="caps"> THEN </span> <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> OLD </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> READ </span> ’, <span class="caps"> IOSTAT </span> =ierror) <span class="caps"> ELSEIF </span> ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) <span class="caps"> THEN </span> ! there is no append for restart-files. They are NEW ! <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> NEW </span> ’, &amp; <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> UNKNOWN </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> WRITE </span> ’, <span class="caps"> IOSTAT </span> =ierror) <p> <br/> and compile a new binary. <br/> This generally changed in the climate version, but not yet done for the “fresh” release of the version cosmo5.0 </p>
BR replied

Please check, if the file 7780OUT_DIR/restarts/lrfd2010051100o already exists.
If so, either delete the file or better:
In io_utilities.f90 in open_file around about line 770 change ‘ NEW ’ to ‘ UNKNOWN ’ for the file opening 



CASE (‘bina’)

IF (my_id 0) THEN IF (ymode(1:1) ‘r’) THEN OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ OLD ’, & ACTION =‘ READ ’, IOSTAT =ierror) ELSEIF ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) THEN ! there is no append for restart-files. They are NEW ! OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ NEW ’, & OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ UNKNOWN ’, & ACTION =‘ WRITE ’, IOSTAT =ierror)


and compile a new binary.
This generally changed in the climate version, but not yet done for the “fresh” release of the version cosmo5.0

View in channel
<p> The file lrfd2010051100o doesn’t exist. But, it seems to me, the problem could be somewhere else, because we have run this compiled binary file previously without any problems. Maybe, I didn’t understand something, but I have run the <span class="caps"> COSMO </span> 5 climate version, not the <span class="caps"> NWP </span> one. I have looked into io_utilities.f90 file, the following states there: </p> <p> <pre><br/> CASE (‘bina’)</pre> </p> IF (my_id 0) THEN IF (ymode(1:1) ‘r’) <span class="caps"> THEN </span> <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> OLD </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> READ </span> ’, <span class="caps"> IOSTAT </span> =ierror) <span class="caps"> ELSEIF </span> ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) <span class="caps"> THEN </span> ! there is no append for restart-files. They are NEW <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> NEW </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> WRITE </span> ’, <span class="caps"> IOSTAT </span> =ierror) <p> As I understand, this is correctly… </p>

  @vladimirplatonov in #9d30e69

<p> The file lrfd2010051100o doesn’t exist. But, it seems to me, the problem could be somewhere else, because we have run this compiled binary file previously without any problems. Maybe, I didn’t understand something, but I have run the <span class="caps"> COSMO </span> 5 climate version, not the <span class="caps"> NWP </span> one. I have looked into io_utilities.f90 file, the following states there: </p> <p> <pre><br/> CASE (‘bina’)</pre> </p> IF (my_id 0) THEN IF (ymode(1:1) ‘r’) <span class="caps"> THEN </span> <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> OLD </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> READ </span> ’, <span class="caps"> IOSTAT </span> =ierror) <span class="caps"> ELSEIF </span> ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) <span class="caps"> THEN </span> ! there is no append for restart-files. They are NEW <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> NEW </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> WRITE </span> ’, <span class="caps"> IOSTAT </span> =ierror) <p> As I understand, this is correctly… </p>
VP replied

The file lrfd2010051100o doesn’t exist. But, it seems to me, the problem could be somewhere else, because we have run this compiled binary file previously without any problems. Maybe, I didn’t understand something, but I have run the COSMO 5 climate version, not the NWP one. I have looked into io_utilities.f90 file, the following states there: 



CASE (‘bina’)

IF (my_id 0) THEN IF (ymode(1:1) ‘r’) THEN OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ OLD ’, & ACTION =‘ READ ’, IOSTAT =ierror) ELSEIF ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) THEN ! there is no append for restart-files. They are NEW OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ NEW ’, & ACTION =‘ WRITE ’, IOSTAT =ierror)

As I understand, this is correctly…

View in channel
<p> I have set nhour_restart parameter to hstop value. Just all simulation was OK, except the last step before writing the restart file. I have remarked in log-file (attached as slurm-722843.out), the lbfd….nc file is opening and closing at every step, but there are no any lbfd…nc files in any directories. After closing these files the string <br/> <pre> All variables and levels are read for boundary data </pre> <br/> was written. Also, the <span class="caps"> OUTPUT </span> log file wasn’t written (it is empty)… Maybe, it all could be associated with some <span class="caps"> NAMELIST </span> parameters, which I don’t know well… <br/> Thank you for any suggestions and help! </p>

  @vladimirplatonov in #1a5f47b

<p> I have set nhour_restart parameter to hstop value. Just all simulation was OK, except the last step before writing the restart file. I have remarked in log-file (attached as slurm-722843.out), the lbfd….nc file is opening and closing at every step, but there are no any lbfd…nc files in any directories. After closing these files the string <br/> <pre> All variables and levels are read for boundary data </pre> <br/> was written. Also, the <span class="caps"> OUTPUT </span> log file wasn’t written (it is empty)… Maybe, it all could be associated with some <span class="caps"> NAMELIST </span> parameters, which I don’t know well… <br/> Thank you for any suggestions and help! </p>
VP replied

I have set nhour_restart parameter to hstop value. Just all simulation was OK, except the last step before writing the restart file. I have remarked in log-file (attached as slurm-722843.out), the lbfd….nc file is opening and closing at every step, but there are no any lbfd…nc files in any directories. After closing these files the string 

All variables and levels are read for boundary data

was written. Also, the OUTPUT log file wasn’t written (it is empty)… Maybe, it all could be associated with some NAMELIST parameters, which I don’t know well…
Thank you for any suggestions and help!

View in channel
<p> From the log file the file <br/> <code> /mnt/msu/users/vplatonov/COSMO_CLM/CLM/EXPERIMENTS/ETR/summer2010test_downscaling/LM/lbfd2010083118.nc </code> <br/> does either not exist or has wrong permissions at the time when <span class="caps"> COSMO </span> was executed. <br/> Regarding restarts please change your source code according to <br/> <pre><br/> CASE (‘bina’)</pre> </p> IF (my_id 0) THEN IF (ymode(1:1) ‘r’) <span class="caps"> THEN </span> <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> OLD </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> READ </span> ’, <span class="caps"> IOSTAT </span> =ierror) <span class="caps"> ELSEIF </span> ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) <span class="caps"> THEN </span> ! there is no append for restart-files. They are NEW ! <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> NEW </span> ’, &amp; <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> UNKNOWN </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> WRITE </span> ’, <span class="caps"> IOSTAT </span> =ierror) <p> <br/> otherwise you run into trouble in future. <br/> Make always sure that I/O directories and necessary input files exist and have read/write permission. </p>

  @burkhardtrockel in #792ecd3

<p> From the log file the file <br/> <code> /mnt/msu/users/vplatonov/COSMO_CLM/CLM/EXPERIMENTS/ETR/summer2010test_downscaling/LM/lbfd2010083118.nc </code> <br/> does either not exist or has wrong permissions at the time when <span class="caps"> COSMO </span> was executed. <br/> Regarding restarts please change your source code according to <br/> <pre><br/> CASE (‘bina’)</pre> </p> IF (my_id 0) THEN IF (ymode(1:1) ‘r’) <span class="caps"> THEN </span> <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> OLD </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> READ </span> ’, <span class="caps"> IOSTAT </span> =ierror) <span class="caps"> ELSEIF </span> ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) <span class="caps"> THEN </span> ! there is no append for restart-files. They are NEW ! <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> NEW </span> ’, &amp; <span class="caps"> OPEN </span> (nudat, <span class="caps"> FILE </span> = <acronym title="datname"> <span class="caps"> TRIM </span> </acronym> , <span class="caps"> FORM </span> =‘ <span class="caps"> UNFORMATTED </span> ’, <span class="caps"> STATUS </span> =‘ <span class="caps"> UNKNOWN </span> ’, &amp; <span class="caps"> ACTION </span> =‘ <span class="caps"> WRITE </span> ’, <span class="caps"> IOSTAT </span> =ierror) <p> <br/> otherwise you run into trouble in future. <br/> Make always sure that I/O directories and necessary input files exist and have read/write permission. </p>
BR replied

From the log file the file
/mnt/msu/users/vplatonov/COSMO_CLM/CLM/EXPERIMENTS/ETR/summer2010test_downscaling/LM/lbfd2010083118.nc
does either not exist or has wrong permissions at the time when COSMO was executed.
Regarding restarts please change your source code according to 



CASE (‘bina’)

IF (my_id 0) THEN IF (ymode(1:1) ‘r’) THEN OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ OLD ’, & ACTION =‘ READ ’, IOSTAT =ierror) ELSEIF ((ymode(1:1) 'w') .OR. (ymode(1:1) ‘a’)) THEN ! there is no append for restart-files. They are NEW ! OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ NEW ’, & OPEN (nudat, FILE = TRIM , FORM =‘ UNFORMATTED ’, STATUS =‘ UNKNOWN ’, & ACTION =‘ WRITE ’, IOSTAT =ierror)


otherwise you run into trouble in future.
Make always sure that I/O directories and necessary input files exist and have read/write permission.

View in channel