MPI nodes all return 0 from pc.id()

[Helveg]

When mpi4py is imported before h.nrnmpi_init() is called, all nodes think they are node 0.

This might be because mpi4py calls MPI.Init() and MPI.Init_thread if MPI hasn't been initialized at that point.

Reproducers

nrnmpi_init first (IDs OK)

from neuron import h

h.nrnmpi_init()
pc = h.ParallelContext()

from mpi4py import MPI

print("I am MPI rank", MPI.COMM_WORLD.rank, "but have ParallelContext ID", pc.id())

h.quit()

MPI Init first (IDs not OK)

from neuron import h
from mpi4py import MPI

h.nrnmpi_init()
pc = h.ParallelContext()


print("I am MPI rank", MPI.COMM_WORLD.rank, "but have ParallelContext ID", pc.id())

h.quit()

[ramcdougal]

It's worse than that.

Prior to 7.7, there was no h.nrnmpi_init() and the pure Python way of initializing MPI was to import mpi4py... I tested on both Linux and Mac, and this is no longer works... looks like you currently have to use h.nrnmpi_init().

[nrnhines]

For now the work around is to call h.nrnmpi_init() prior to import mpi4py.
The logic has become incrementally convoluted and needs to be regularized. Prior to h.nrnmpi_init
when launching python one could either import mpi4py or have set the environment variable
NEURON_INIT_MPI. When launching nrniv one had to use the argument -mpi. A
recent change for a python launch was export NEURON_INIT_MPI=0 meant do NOT load libmpi dynamically (let alone initialise it) to avoid a crash on a certain machine #258 . And the most recent change was to fix an edge case for that #427 . The dimensions are
launch python or nrniv
mpi does or does not exist on the machine
mpi4py imported or not
mpi statically or dynamically linked to nrniv or dynamically loaded after launch.
NEURON_INIT_MPI does not exist, or exists with a value of 0 or 1
User calls h.nrnmpi_init() or not.
Presently you can run your second fragment correctly with

hines@hines-T7500:~/tmp$ export NEURON_INIT_MPI=1
hines@hines-T7500:~/tmp$ mpiexec -n 4 python temp2.py
numprocs=4
I am MPI rank 0 but have ParallelContext ID 0
I am MPI rank 3 but have ParallelContext ID 3
I am MPI rank 2 but have ParallelContext ID 2
I am MPI rank 1 but have ParallelContext ID 1

I have to decide whether to fix just this issue or whether to refactor the current implementation.

[nrnhines]

Pull #431 request fixes all the issues mentioned in in this issue but does have some strangeness.
The good news is that fragment 1 (temp1.py) and fragment 2 (temp2.py) above, do work properly with

mpiexec -n 2 python temp1.py
mpiexec -n 2 python temp2.py
mpiexec -n 2 nrniv -mpi -python temp1.py
mpiexec -n 2 nrniv -mpi -python temp2.py

but if you leave off the -mpi for the nrniv launches, you will get two banners unless launching with the -nobanner option. Furthermore, if you launch without mpiexec and execute h.nrnmpi_init()
it will print 'numprocs=1' but 'from neuron import gui' will also work.

[ramcdougal]

@adamjhn pointed out that pre-7.7 the rule was that mpi4py had to be imported before neuron. If you do that, then ParallelContext gets the right ids... so everything that used to work still works, and my previous comment was misleading.

[nrnhines]

Ok. I don't see where the documentation mentions an order issue between mpi4py and nrnmpi_init()
It only cautions that h.nrnmpi_init() has to be executed before the first instantiation of ParallelContext. It is only necessary if you do not import mpi4py and you do not
export NEURON_INIT_MPI=1
(Note the clause about PVM needs to be removed) I think that if you rely on mpi4py alone, then it
must be imported prior to neuron. #431 can be construed as a minor change that allows one
to import neuron, then import mpi4py, then execute neuron.h.nrnmpi_init(). Prior to #431 a call to the last would have been a noop.

[Helveg]

@adamjhn pointed out that pre-7.7 the rule was that mpi4py had to be imported before neuron. If you do that, then ParallelContext gets the right ids... so everything that used to work still works, and my previous comment was misleading.

I'd also like to suggest not only placing a note in the documentation for cases like this or merely agreeing on an unwritten rule that one should somehow be aware of, but to throw an exception. You cannot expect the user to read through all available documentation resources entirely before using any part of NEURON and to remember every footnote ad eternum.

[Helveg]

Hi @nrnhines , I'm faced with an environment where I have to use NEURON 7.6 and nrnmpi_init is not available. What exactly is the correct procedure for MPI initialisation there?

Currently I use:

    def _init_pc(self):
        if not hasattr(self, "_PythonHocInterpreter__pc"):
            self.__h.nrnmpi_init()
            self.__pc = ParallelContext(self, self.__h.ParallelContext())

    @property
    def parallel(self):
        self._init_pc()
        return self.__pc

I'd like to use a try/except block around the self.__h.nrnmpi_init() call to catch the missing h.nrnmpi_init but I don't know what I'd have to do there in NEURON 7.6

[ramcdougal]

import mpi4py

at the very beginning, before you do any NEURON imports. That works in all recent versions (but requires mpi4py to be installed).

Alternatively, you can force NEURON to do the initialization at import by setting the environment variable NEURON_INIT_MPI=1 .

[Helveg]

import mpi4py

at the very beginning, before you do any NEURON imports. That works in all recent versions (but requires mpi4py to be installed).

Alternatively, you can force NEURON to do the initialization at import by setting the environment variable NEURON_INIT_MPI=1 .

But don't my snippets show that importing mpi4py first doesn't work? (Edit: no sorry, neuron is always imported first, my bad)

Since I don't control either of those options as a library author, is there a way to detect whether NEURON's MPI is OK or not? I can throw an exception with some explanation if it isn't.

[ramcdougal]

It looks like things are fine if and only if MPI.COMM_WORLD.size == pc.nhost().

If they're not equal, I would have the exception advise moving the import of mpi4py to the very beginning of the program.

[pramodk]

@Helveg : do you still have any issues or questions regarding this ticket? Thanks!

[Helveg]

I still sometimes encounter verbose MPI errors about ORTE or OPAL or init things so it's still seems "sensitive" to the machine environment, but generally no: the standard flow on most machines seems robust enough.

[pramodk]

Thank you @Helveg for confirming!

I still sometimes encounter verbose MPI errors about ORTE or OPAL or init things so it's still seems "sensitive" to the machine environment,

I have seen those as well, typically with OpenMPI. These are related to MPI libraries on specific platform and not directly related to NEURON itself. If you see any issues or have any questions, please report it here and will be happy to help.

I will close this ticket then.