CCLM compiled with IntelMPI – in #9: CCLM

in #9: CCLM

<p> Hi all, </p> <p> <span class="caps"> CCLM </span> was working fine using openMPI. However, I started having problems due to some missing libraries of OpenMPI in the computer cluster. While this getting fixed, I have used intelMPI to compile <span class="caps"> CCLM </span> model but having problems during execution. Has anyone in the community been able to use intelMPI with the <span class="caps"> CCLM </span> model? I would guess that yes and shouldn’t be a problem but I am not sure why it doesn’t work here. Following is the error message. </p> <pre> Fatal error in MPI_Send: Other MPI error, error stack: MPI_Send(186): MPI_Send(buf=0x298eff0, count=5, MPI_INTEGER, dest=0, tag=6665, comm=0x84000003) failed MPID_Send(52): DEADLOCK: attempting to send a message to the local process without a prior matching receive </pre> <p> Thanks in advance, <br/> Cecille </p>

  @redc_migration in #1704301

<p> Hi all, </p> <p> <span class="caps"> CCLM </span> was working fine using openMPI. However, I started having problems due to some missing libraries of OpenMPI in the computer cluster. While this getting fixed, I have used intelMPI to compile <span class="caps"> CCLM </span> model but having problems during execution. Has anyone in the community been able to use intelMPI with the <span class="caps"> CCLM </span> model? I would guess that yes and shouldn’t be a problem but I am not sure why it doesn’t work here. Following is the error message. </p> <pre> Fatal error in MPI_Send: Other MPI error, error stack: MPI_Send(186): MPI_Send(buf=0x298eff0, count=5, MPI_INTEGER, dest=0, tag=6665, comm=0x84000003) failed MPID_Send(52): DEADLOCK: attempting to send a message to the local process without a prior matching receive </pre> <p> Thanks in advance, <br/> Cecille </p>

CCLM compiled with IntelMPI

re wrote

Hi all,

CCLM was working fine using openMPI. However, I started having problems due to some missing libraries of OpenMPI in the computer cluster. While this getting fixed, I have used intelMPI to compile CCLM model but having problems during execution. Has anyone in the community been able to use intelMPI with the CCLM model? I would guess that yes and shouldn’t be a problem but I am not sure why it doesn’t work here. Following is the error message. 

Fatal error in MPI_Send: Other MPI error, error stack:
MPI_Send(186): MPI_Send(buf=0x298eff0, count=5, MPI_INTEGER, dest=0, tag=6665, comm=0x84000003) failed
MPID_Send(52): DEADLOCK: attempting to send a message to the local process without a prior matching receive

Thanks in advance,
Cecille

View in channel
<p> Your question regarding intelMPI has not been send due to RedC Email problems. <br/> I hope it works now and someone of the <span class="caps"> CLM </span> -Community has already tested intelMPI. I have not, sorry! </p>

  @burkhardtrockel in #4ffafea

<p> Your question regarding intelMPI has not been send due to RedC Email problems. <br/> I hope it works now and someone of the <span class="caps"> CLM </span> -Community has already tested intelMPI. I have not, sorry! </p>
BR replied

Your question regarding intelMPI has not been send due to RedC Email problems.
I hope it works now and someone of the CLM -Community has already tested intelMPI. I have not, sorry!

View in channel
<p> Which “computer cluster”? <br/> I “played” a little bit with the <span class="caps"> INTELMPI </span> at the new machine “ <span class="caps"> MISTRAL </span> ” at <span class="caps"> DKRZ </span> in Hamburg. <br/> It worked (however, I decided for myself to use the <span class="caps"> BULLMPI </span> ). <br/> I don’t have any experience with <span class="caps"> INTELMPI </span> on other machines. </p> <p> Hans-Jürgen </p>

  @hans-jürgenpanitz in #51b1345

<p> Which “computer cluster”? <br/> I “played” a little bit with the <span class="caps"> INTELMPI </span> at the new machine “ <span class="caps"> MISTRAL </span> ” at <span class="caps"> DKRZ </span> in Hamburg. <br/> It worked (however, I decided for myself to use the <span class="caps"> BULLMPI </span> ). <br/> I don’t have any experience with <span class="caps"> INTELMPI </span> on other machines. </p> <p> Hans-Jürgen </p>
HP replied

Which “computer cluster”?
I “played” a little bit with the INTELMPI at the new machine “ MISTRAL ” at DKRZ in Hamburg.
It worked (however, I decided for myself to use the BULLMPI ).
I don’t have any experience with INTELMPI on other machines.

Hans-Jürgen

View in channel
<p> Hi Hans-Jürgen </p> <p> Thanks for sharing your experiences with using <span class="caps"> INTELMPI </span> at Mistral. </p> <p> I guess that you were using <span class="caps"> COSMO </span> 5.0 for performing these tests? </p> <p> Regards </p> <p> Andrew </p>

  @redc_migration in #e574b3b

<p> Hi Hans-Jürgen </p> <p> Thanks for sharing your experiences with using <span class="caps"> INTELMPI </span> at Mistral. </p> <p> I guess that you were using <span class="caps"> COSMO </span> 5.0 for performing these tests? </p> <p> Regards </p> <p> Andrew </p>
re replied

Hi Hans-Jürgen

Thanks for sharing your experiences with using INTELMPI at Mistral.

I guess that you were using COSMO 5.0 for performing these tests?

Regards

Andrew

View in channel
<p> Hi Andrew, </p> <p> correct. I used COMSO_5.0. <br/> I reported on these tests during the Assembly in Luxmebourg. <br/> My recommendation was to use the BULL-MPI since it was somewhat faster than the INTEL-MPI, especially when using larger number of nodes (&gt; 25 to 30). <br/> Nevertheless, <span class="caps"> INTEL </span> - <span class="caps"> MPI </span> worked. </p> <p> Of course, in the meantime I also applied cosmo_4.8_clm17/19 on <span class="caps"> MISTRAL </span> (sucessfully), but only with <span class="caps"> BULL </span> - <span class="caps"> MPI </span> (I myself decided to stick on <span class="caps"> BULL </span> - <span class="caps"> MPI </span> ). </p> <p> Hans-Jürgen </p>

  @hans-jürgenpanitz in #ec59b61

<p> Hi Andrew, </p> <p> correct. I used COMSO_5.0. <br/> I reported on these tests during the Assembly in Luxmebourg. <br/> My recommendation was to use the BULL-MPI since it was somewhat faster than the INTEL-MPI, especially when using larger number of nodes (&gt; 25 to 30). <br/> Nevertheless, <span class="caps"> INTEL </span> - <span class="caps"> MPI </span> worked. </p> <p> Of course, in the meantime I also applied cosmo_4.8_clm17/19 on <span class="caps"> MISTRAL </span> (sucessfully), but only with <span class="caps"> BULL </span> - <span class="caps"> MPI </span> (I myself decided to stick on <span class="caps"> BULL </span> - <span class="caps"> MPI </span> ). </p> <p> Hans-Jürgen </p>
HP replied

Hi Andrew,

correct. I used COMSO_5.0.
I reported on these tests during the Assembly in Luxmebourg.
My recommendation was to use the BULL-MPI since it was somewhat faster than the INTEL-MPI, especially when using larger number of nodes (> 25 to 30).
Nevertheless, INTEL - MPI worked.

Of course, in the meantime I also applied cosmo_4.8_clm17/19 on MISTRAL (sucessfully), but only with BULL - MPI (I myself decided to stick on BULL - MPI ).

Hans-Jürgen

View in channel
<p> Thank you Hans-Jürgen and Andrew for your reply. I am running cosmo4.8_clm17 in Tier-1 supercomputer at Cenaero. However, you only used <span class="caps"> BULL </span> - <span class="caps"> MPI </span> for this model version and did not tested IntelMPI. I would guess that if it worked with COSMO5.0 then it should work as well for previous model versions. But I would like to hear from other model users if previous model versions were able to work with IntelMPI. From what I know so far, <span class="caps"> BULL </span> - <span class="caps"> MPI </span> is not available in this supercomputer. Thank you again. </p> <p> Cecille </p>

  @redc_migration in #dc4cff0

<p> Thank you Hans-Jürgen and Andrew for your reply. I am running cosmo4.8_clm17 in Tier-1 supercomputer at Cenaero. However, you only used <span class="caps"> BULL </span> - <span class="caps"> MPI </span> for this model version and did not tested IntelMPI. I would guess that if it worked with COSMO5.0 then it should work as well for previous model versions. But I would like to hear from other model users if previous model versions were able to work with IntelMPI. From what I know so far, <span class="caps"> BULL </span> - <span class="caps"> MPI </span> is not available in this supercomputer. Thank you again. </p> <p> Cecille </p>
re replied

Thank you Hans-Jürgen and Andrew for your reply. I am running cosmo4.8_clm17 in Tier-1 supercomputer at Cenaero. However, you only used BULL - MPI for this model version and did not tested IntelMPI. I would guess that if it worked with COSMO5.0 then it should work as well for previous model versions. But I would like to hear from other model users if previous model versions were able to work with IntelMPI. From what I know so far, BULL - MPI is not available in this supercomputer. Thank you again.

Cecille

View in channel
<p> Hello Cecille, </p> <p> I am using intel <span class="caps"> MPI </span> since older version of <span class="caps"> COSMO </span> up to 5.01 and never had a problem. I am also have openMPI installed in the machine I use (Dell Power <span class="caps"> EDGE </span> X64, 652 <span class="caps"> CPU </span> s) and I also have used <br/> it without a problem. So, yes Intel <span class="caps"> MPI </span> can run any version of <span class="caps"> COSMO </span> . Now, looking into your problem, I believe it is a matter of setting the environment variable correctly. But, it all depends <br/> of which version of Intel <span class="caps"> MPI </span> are you using, since some versions can allow more compatibility with your older openMPI env than others. Buffering, can be one of such incompatibility and then execution is <br/> halted with error if the comm library doesn’t recon the sending. Usually, “export <span class="caps"> MPI </span> _COMPATIBILITY=#” , before mpiifort or mpirun, will make it through, where # is either 3 or 4, depending <br/> on the intel <span class="caps"> MPI </span> version. Probably, you need to start over with the right environment for intel (you might have a look on <span class="caps"> WRF </span> ’s forum to see what comes out from .configure for intel). <br/> Best wishes, <br/> Reinaldo </p>

  @reinaldosilveira in #9e13b90

<p> Hello Cecille, </p> <p> I am using intel <span class="caps"> MPI </span> since older version of <span class="caps"> COSMO </span> up to 5.01 and never had a problem. I am also have openMPI installed in the machine I use (Dell Power <span class="caps"> EDGE </span> X64, 652 <span class="caps"> CPU </span> s) and I also have used <br/> it without a problem. So, yes Intel <span class="caps"> MPI </span> can run any version of <span class="caps"> COSMO </span> . Now, looking into your problem, I believe it is a matter of setting the environment variable correctly. But, it all depends <br/> of which version of Intel <span class="caps"> MPI </span> are you using, since some versions can allow more compatibility with your older openMPI env than others. Buffering, can be one of such incompatibility and then execution is <br/> halted with error if the comm library doesn’t recon the sending. Usually, “export <span class="caps"> MPI </span> _COMPATIBILITY=#” , before mpiifort or mpirun, will make it through, where # is either 3 or 4, depending <br/> on the intel <span class="caps"> MPI </span> version. Probably, you need to start over with the right environment for intel (you might have a look on <span class="caps"> WRF </span> ’s forum to see what comes out from .configure for intel). <br/> Best wishes, <br/> Reinaldo </p>
RS replied

Hello Cecille,

I am using intel MPI since older version of COSMO up to 5.01 and never had a problem. I am also have openMPI installed in the machine I use (Dell Power EDGE X64, 652 CPU s) and I also have used
it without a problem. So, yes Intel MPI can run any version of COSMO . Now, looking into your problem, I believe it is a matter of setting the environment variable correctly. But, it all depends
of which version of Intel MPI are you using, since some versions can allow more compatibility with your older openMPI env than others. Buffering, can be one of such incompatibility and then execution is
halted with error if the comm library doesn’t recon the sending. Usually, “export MPI _COMPATIBILITY=#” , before mpiifort or mpirun, will make it through, where # is either 3 or 4, depending
on the intel MPI version. Probably, you need to start over with the right environment for intel (you might have a look on WRF ’s forum to see what comes out from .configure for intel).
Best wishes,
Reinaldo

View in channel
<p> Hello again Cecille, <br/> I was looking at my previous comment and it might be confusing. In fact, you will need to tell the loader which mpirun are you using, <br/> before running the scripts, otherwise it may look at the wrong environment variables. For example, in my case there is a small script in the <br/> /opt/intel/impi/4.1.1/bin64/mpivars.sh that set the I_MPI_ROOT and then the other variables and the libraries. They can be compatible with openMPI variables, which doesn’t look like in your case, and nothing needs to be done. But, when it is not the case, you need to <br/> load such script (eg. “source /opt/intel/impi/4.1.1/bin64/mpivars.sh”) as well as any other related library. Exporting <span class="caps"> MPI </span> _COMPATIBILIY might <br/> not work. <br/> Reinaldo </p>

  @reinaldosilveira in #446421c

<p> Hello again Cecille, <br/> I was looking at my previous comment and it might be confusing. In fact, you will need to tell the loader which mpirun are you using, <br/> before running the scripts, otherwise it may look at the wrong environment variables. For example, in my case there is a small script in the <br/> /opt/intel/impi/4.1.1/bin64/mpivars.sh that set the I_MPI_ROOT and then the other variables and the libraries. They can be compatible with openMPI variables, which doesn’t look like in your case, and nothing needs to be done. But, when it is not the case, you need to <br/> load such script (eg. “source /opt/intel/impi/4.1.1/bin64/mpivars.sh”) as well as any other related library. Exporting <span class="caps"> MPI </span> _COMPATIBILIY might <br/> not work. <br/> Reinaldo </p>
RS replied

Hello again Cecille,
I was looking at my previous comment and it might be confusing. In fact, you will need to tell the loader which mpirun are you using,
before running the scripts, otherwise it may look at the wrong environment variables. For example, in my case there is a small script in the
/opt/intel/impi/4.1.1/bin64/mpivars.sh that set the I_MPI_ROOT and then the other variables and the libraries. They can be compatible with openMPI variables, which doesn’t look like in your case, and nothing needs to be done. But, when it is not the case, you need to
load such script (eg. “source /opt/intel/impi/4.1.1/bin64/mpivars.sh”) as well as any other related library. Exporting MPI _COMPATIBILIY might
not work.
Reinaldo

View in channel