Long term simulation – in #12: CCLM Starter Package Support

in #12: CCLM Starter Package Support

<p> Hello, <br/> I am attempting running a long term <span class="caps"> CORDEX </span> simulation using 11 nodes with 32 processors per node. The job runs successfully, but periodically stops during the post processing stage. This problem may possibly be indicating a conflict due to the fact that <span class="caps"> CCLM </span> job is not yet released from the queue. My feeling is that problem is more complex however since I get the same results when I resubmit the post job after several hours. Still, after repeating this action several times without any change, I am finally able to continue the run. <br/> I have experienced the problem several times already. Please let me know your recommendations. I can add that I have not had (or not noticed) such problem in my previous runs using a smaller number of nodes. <br/> Simon </p>

  @redc_migration in #e350e2f

<p> Hello, <br/> I am attempting running a long term <span class="caps"> CORDEX </span> simulation using 11 nodes with 32 processors per node. The job runs successfully, but periodically stops during the post processing stage. This problem may possibly be indicating a conflict due to the fact that <span class="caps"> CCLM </span> job is not yet released from the queue. My feeling is that problem is more complex however since I get the same results when I resubmit the post job after several hours. Still, after repeating this action several times without any change, I am finally able to continue the run. <br/> I have experienced the problem several times already. Please let me know your recommendations. I can add that I have not had (or not noticed) such problem in my previous runs using a smaller number of nodes. <br/> Simon </p>

Long term simulation

re wrote

Hello,
I am attempting running a long term CORDEX simulation using 11 nodes with 32 processors per node. The job runs successfully, but periodically stops during the post processing stage. This problem may possibly be indicating a conflict due to the fact that CCLM job is not yet released from the queue. My feeling is that problem is more complex however since I get the same results when I resubmit the post job after several hours. Still, after repeating this action several times without any change, I am finally able to continue the run.
I have experienced the problem several times already. Please let me know your recommendations. I can add that I have not had (or not noticed) such problem in my previous runs using a smaller number of nodes.
Simon

View in channel
<p> Running the post processing before the <span class="caps"> CCLM </span> job has been released can actually lead to problems. On Blizzard I experienced such a problem and therefore set in the post processing script: <br/> <code> sleep 60 # to avoid conflict if CCLM job is not yet released from the queue (may not be relevant on all systems) </code> <br/> You may find this line when looking at the template scripts. <br/> However, if I understand you right, you re-submitted the post processing job hours later individually and it happened again. This is really strange. It may be due to some problems in your computing system. This can be complex to find. A brute approach to narrow this down to the line where it happens in the script is to insert an <code> echo test nn </code> after each line. </p>

  @burkhardtrockel in #310b87e

<p> Running the post processing before the <span class="caps"> CCLM </span> job has been released can actually lead to problems. On Blizzard I experienced such a problem and therefore set in the post processing script: <br/> <code> sleep 60 # to avoid conflict if CCLM job is not yet released from the queue (may not be relevant on all systems) </code> <br/> You may find this line when looking at the template scripts. <br/> However, if I understand you right, you re-submitted the post processing job hours later individually and it happened again. This is really strange. It may be due to some problems in your computing system. This can be complex to find. A brute approach to narrow this down to the line where it happens in the script is to insert an <code> echo test nn </code> after each line. </p>
BR replied

Running the post processing before the CCLM job has been released can actually lead to problems. On Blizzard I experienced such a problem and therefore set in the post processing script:
sleep 60 # to avoid conflict if CCLM job is not yet released from the queue (may not be relevant on all systems)
You may find this line when looking at the template scripts.
However, if I understand you right, you re-submitted the post processing job hours later individually and it happened again. This is really strange. It may be due to some problems in your computing system. This can be complex to find. A brute approach to narrow this down to the line where it happens in the script is to insert an echo test nn after each line.

View in channel
<p> I saw the sleep 60 line you mentioned of course. <br/> I have even changed the value to 240. This does not help however. <br/> In my runs when the post job stops DATE1 is equal to DATE2. <br/> And in the runs ending successfully the two values differ. I do not know yet why this happens, <br/> but as a temporary brutal solution I just have commented the line let “ <span class="caps"> SEC </span> _CHECK=DATE2-DATE1” and set another one <span class="caps"> SEC </span> _CHECK=1 instead. <br/> The job runs now but there may be other problems due to the change. <br/> If I understand the code correctly the check is just to be sure that there were enough data files, so probably on our machine my correction will work good? </p>

  @redc_migration in #e22a7fb

<p> I saw the sleep 60 line you mentioned of course. <br/> I have even changed the value to 240. This does not help however. <br/> In my runs when the post job stops DATE1 is equal to DATE2. <br/> And in the runs ending successfully the two values differ. I do not know yet why this happens, <br/> but as a temporary brutal solution I just have commented the line let “ <span class="caps"> SEC </span> _CHECK=DATE2-DATE1” and set another one <span class="caps"> SEC </span> _CHECK=1 instead. <br/> The job runs now but there may be other problems due to the change. <br/> If I understand the code correctly the check is just to be sure that there were enough data files, so probably on our machine my correction will work good? </p>
re replied

I saw the sleep 60 line you mentioned of course.
I have even changed the value to 240. This does not help however.
In my runs when the post job stops DATE1 is equal to DATE2.
And in the runs ending successfully the two values differ. I do not know yet why this happens,
but as a temporary brutal solution I just have commented the line let “ SEC _CHECK=DATE2-DATE1” and set another one SEC _CHECK=1 instead.
The job runs now but there may be other problems due to the change.
If I understand the code correctly the check is just to be sure that there were enough data files, so probably on our machine my correction will work good?

View in channel
<p> The line <code> let "SEC_CHECK=DATE2-DATE1" </code> counts how much time the checking process needs. This is just for information and not necessary for the post-processing of the data. Therefore setting it to <code> SEC_CHECK=1 </code> does not matter. <br/> I experienced some problem with a similar command sometimes in another script. You may try the following instead of using the <code> let </code> command. <br/> <code> SEC_CHECK=$(python -c "print ${DATE2}-${DATE1}") </code> <br/> Anyway, just setting <code> SEC_CHECK=1 </code> is fine, if you do not need the time information for some reason. </p>

  @burkhardtrockel in #5e4efb2

<p> The line <code> let "SEC_CHECK=DATE2-DATE1" </code> counts how much time the checking process needs. This is just for information and not necessary for the post-processing of the data. Therefore setting it to <code> SEC_CHECK=1 </code> does not matter. <br/> I experienced some problem with a similar command sometimes in another script. You may try the following instead of using the <code> let </code> command. <br/> <code> SEC_CHECK=$(python -c "print ${DATE2}-${DATE1}") </code> <br/> Anyway, just setting <code> SEC_CHECK=1 </code> is fine, if you do not need the time information for some reason. </p>
BR replied

The line let "SEC_CHECK=DATE2-DATE1" counts how much time the checking process needs. This is just for information and not necessary for the post-processing of the data. Therefore setting it to SEC_CHECK=1 does not matter.
I experienced some problem with a similar command sometimes in another script. You may try the following instead of using the let command.
SEC_CHECK=$(python -c "print ${DATE2}-${DATE1}")
Anyway, just setting SEC_CHECK=1 is fine, if you do not need the time information for some reason.

View in channel