Intel compilers require some help

[sbryngelson]

There seems to be some issue with OpenMPI + Intel compilers (I used openmpi/4.0.2-intel20.4 on Bridges2).

Update: same issue below occurs with intelmpi

• Adding add_compile_options(-O1) to cmakelists.txt is required to pass tests
• export FC=ifort; export CC=icc; export CXX=icc; before compiling is required (shouldn't this be picked up automatically by cmake? it otherwise tries to use GNU)

If one removes the optimization flag above, then we get error

   [ 97%]  Time step       50 of 51 @ t_step = 49
   NaN(s) in timestep output.           0           0           0           0
            50          49          39           0
  NaN(s) in timestep output.

for test Test tests/55533234: 2D -> bc=-1...

So, this issue appears not to be related to viscous + bubbles, but rather this is the first 2D test case. I think the confusion comes from the fact that the test immediately before this one is viscous + bubbles, but after double checking that one, it does indeed pass fine. This 2D case quickly throws NaNs.

This issue is related to the comments on PR: #152

[sbryngelson]

From @lee-hyeoksu in the PR:

I tried to figure out what happens in test 555332334 and I found that the NaNs actually occur at the first time step when populating buffers in s_populate_conservative_variables_buffers in m_rhs.fpp.

In this subroutine, global parameters bc_s%beg and bc_s%end are used for s-direction (s = x, y, or z) to populate buffers based on BCs. However, in the subroutine, the values for these parameters are 0, although they should be some negative integers. So the subroutine does not assign appropriate values to q_cons_qp in buffer regions, which leads to NaNs.

bc_x and bc_y become 0 right after returning from s_read_data_files(q_cons_ts(1)%vf) (https://github.com/MFlowCode/MFC/blob/master/src/simulation/p_main.fpp#L185).
Still I am not sure why this happens for intel compilers, so I keep looking into this issue but I just wanted to share. Any suggestions would be appreciated!

[sbryngelson]

One question is whether this issue has existed for previous versions of MFC, or if some newly introduced feature created it. @lee-hyeoksu can you try downloading and testing older MFC versions (like before your viscous+bubbles PR) and see if they pass this test without the add compiler options flag in cmakelists.

[sbryngelson]

From @lee-hyeoksu in the PR:

I tried to figure out what happens in test 555332334 and I found that the NaNs actually occur at the first time step when populating buffers in s_populate_conservative_variables_buffers in m_rhs.fpp.

In this subroutine, global parameters bc_s%beg and bc_s%end are used for s-direction (s = x, y, or z) to populate buffers based on BCs. However, in the subroutine, the values for these parameters are 0, although they should be some negative integers. So the subroutine does not assign appropriate values to q_cons_qp in buffer regions, which leads to NaNs.

bc_x and bc_y become 0 right after returning from s_read_data_files(q_cons_ts(1)%vf) (master/src/simulation/p_main.fpp#L185).
Still I am not sure why this happens for intel compilers, so I keep looking into this issue but I just wanted to share. Any suggestions would be appreciated!

I'm not sure if this is actually an issue or not. You might notice this happening for other/older versions of MFC. This is because of a call to an MPI routine that sets the BCs for each processor (even though this is a 1-rank case). I think @anandrdbz can explain better.

I could be wrong about the details here, though, and this is a bug.

[anandrdbz]

@lee-hyeoksu the case of 0 ranks should be analogous to a periodic boundary condition, wherein you're essentially performing an MPI_SENDRECV from rank 0 to rank 0.

So this issue will occur only when bc_s is -1, which I presume is the test that fails.

If you wish to turn off this feature, you can go to line 186 in src/simulation/m_mpi_proxy.fpp and remove parallel_io from the if condition (change if(num_procs == 1 and parall_io) to if(num_procs == 1) ). Not sure why the parallel_io is added here in the first place

This should produce exactly the same results for all the test cases running on 1 rank. If it fails on 2 ranks, where this can't be circumvented, the issue would pertain to a bug in the mpi bindings with intel compilers.

[anandrdbz]

@lee-hyeoksu A small correction, you would have to change the if condition in pre_process, simulation as well as post_process in the s_mpi_decompose_computational_domain subroutine in m_mpi_proxy.fpp

[hyeoksu-lee]

@anandrdbz I ran test after removing parallel_io from the if condition in m_mpi_proxy.fpp in pre_process, simulation, and post_process as you noted, and segmentation fault occurred for all tests.