nnUNET trainer

[drusmanshams]

I am getting following error while using spine_muscle_adipose_tissue pipeline
AttributeError: Can't pickle local object 'nnUNetTrainerV2.initialize_network..'

traceback is as follows
Traceback (most recent call last):
File "/Users/usman/Comp2Comp/comp2comp/utils/process.py", line 131, in process_3d
pipeline(output_dir=output_dir, model_dir=model_dir)
File "/Users/usman/Comp2Comp/comp2comp/inference_pipeline.py", line 73, in call
output = inference_class(inference_pipeline=self, **output)
File "/Users/usman/Comp2Comp/comp2comp/inference_pipeline.py", line 69, in call
output = inference_class(
File "/Users/usman/Comp2Comp/comp2comp/muscle_adipose_tissue/muscle_adipose_tissue.py", line 96, in call
predict.predict_cases(
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/site-packages/nnunet/inference/predict.py", line 209, in predict_cases
for preprocessed in preprocessing:
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/site-packages/nnunet/inference/predict.py", line 109, in preprocess_multithreaded
pr.start()
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/usman/Downloads/miniconda3/envs/c2c_env/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'nnUNetTrainerV2.initialize_network..'

anyone else had similar problem and how did you manage?

[ITwindCloud]

I am encountering the same problem, but I don't know how to deal with.

[drusmanshams]

@ITwindCloud hi did you manage to sort?

[Vojtech-Benda]

I'm getting the same error with the spine_muscle_adipose_tissue pipeline. Running on Windows 11, through miniforge virtual environment with python 3.9.

This is the error message without doing any changes:

starting prediction...
ERROR PROCESSING outdir/dcm

Traceback (most recent call last):
  File "C:\dev\python-projects\Comp2Comp\comp2comp\utils\process.py", line 131, in process_3d
    pipeline(output_dir=output_dir, model_dir=model_dir)
  File "C:\dev\python-projects\Comp2Comp\comp2comp\inference_pipeline.py", line 73, in __call__
    output = inference_class(inference_pipeline=self, **output)
  File "C:\dev\python-projects\Comp2Comp\comp2comp\inference_pipeline.py", line 69, in __call__
    output = inference_class(
  File "C:\dev\python-projects\Comp2Comp\comp2comp\muscle_adipose_tissue\muscle_adipose_tissue.py", line 96, in __call__
    predict.predict_cases(
  File "C:\miniforge\envs\c2c-env39\lib\site-packages\nnunet\inference\predict.py", line 209, in predict_cases
    for preprocessed in preprocessing:
  File "C:\miniforge\envs\c2c-env39\lib\site-packages\nnunet\inference\predict.py", line 109, in preprocess_multithreaded
    pr.start()
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'nnUNetTrainerV2.initialize_network.<locals>.<lambda>'

C:\dev\python-projects\Comp2Comp>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Using this #136 change as an attempt in nnunet/training/network_training/nnUNetTrainer.py and nnUNetTrainerV2.py.
Replacing ..., lambda x: x, ... with ..., identity, ... results in this error message instead:

starting prediction...
ERROR PROCESSING outdir/dcm

Traceback (most recent call last):
  File "C:\dev\python-projects\Comp2Comp\comp2comp\utils\process.py", line 131, in process_3d
    pipeline(output_dir=output_dir, model_dir=model_dir)
  File "C:\dev\python-projects\Comp2Comp\comp2comp\inference_pipeline.py", line 73, in __call__
    output = inference_class(inference_pipeline=self, **output)
  File "C:\dev\python-projects\Comp2Comp\comp2comp\inference_pipeline.py", line 69, in __call__
    output = inference_class(
  File "C:\dev\python-projects\Comp2Comp\comp2comp\muscle_adipose_tissue\muscle_adipose_tissue.py", line 96, in __call__
    predict.predict_cases(
  File "C:\miniforge\envs\c2c-env39\lib\site-packages\nnunet\inference\predict.py", line 209, in predict_cases
    for preprocessed in preprocessing:
  File "C:\miniforge\envs\c2c-env39\lib\site-packages\nnunet\inference\predict.py", line 109, in preprocess_multithreaded
    pr.start()
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Generic_UNet.__init__.<locals>.<lambda>' **<- this is different**

C:\dev\python-projects\Comp2Comp>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\miniforge\envs\c2c-env39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

[aedendavis]

I am running into the same issue. Did you ever find a solution?

[ITwindCloud]

@ITwindCloud hi did you manage to sort?

Hi. I added the following codes into 'envs\c2c_env\Lib\site-packages\nnunet\inference\predict.py' to process the multiprocessing of the windows system:

I hope this can help you!

[ITwindCloud]

I am running into the same issue. Did you ever find a solution?

You can refer to my replies above. I hope this can help you too!