compatibility with MONAI v0.6.0 (15/July)

[wyli]

Is your feature request related to a problem? Please describe.
MONAI v0.6.0rc1 is available:
https://pypi.org/project/monai/0.6.0rc1/

this ticket tracks any issues with the new version of MONAI and verifies that the new features are usable:

• load MMAR pretrained https://github.com/Project-MONAI/MONAI/blob/0.6.0rc1/monai/apps/mmars/mmars.py#L204-L207
• dynamic datalist
  there are breaking changes since 0.5.x, we'll release demos Update all the examples & notebooks to use new APIs (14/July) MONAI#2402

cc @Nic-Ma @yiheng-wang-nv

[SachidanandAlle]

Lets verify all basic functionality using 0.6 version..
lets not implement anything for mmar apis (we continue using download apis from monai to download pretrained models)

we will revisit to use the new apis in next version of monailabel (due to time constraints)

[diazandr3s]

I found this error when using monai==0.6.0rc1

GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

Apparently, it is something related to my GPU card (GeForce 3090). Solved this problem by running:

conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Taken from pytorch/pytorch#46794

However, I still get the error when loading a pre-trained model:

Traceback (most recent call last):
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in call
return await self.app(scope, receive, send)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/fastapi/applications.py", line 201, in call
await super().call(scope, receive, send) # pragma: no cover
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 181, in call
raise exc from None
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in call
await self.app(scope, receive, _send)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/exceptions.py", line 82, in call
raise exc from None
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in call
await self.app(scope, receive, sender)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/routing.py", line 580, in call
await route.handle(scope, receive, send)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/routing.py", line 241, in handle
await self.app(scope, receive, send)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/starlette/routing.py", line 52, in app
response = await func(request)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/fastapi/routing.py", line 215, in app
dependant=dependant, values=values, is_coroutine=is_coroutine
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
return await dependant.call(**values)
File "/home/adp20local/Downloads/MONAILabel/monailabel/endpoints/infer.py", line 129, in run_inference
result = instance.infer(request)
File "/home/adp20local/Downloads/MONAILabel/monailabel/interfaces/app.py", line 124, in infer
result_file_name, result_json = task(request)
File "/home/adp20local/Downloads/MONAILabel/monailabel/interfaces/tasks/infer.py", line 198, in call
data = self.run_inferer(data, device=device)
File "/home/adp20local/Downloads/MONAILabel/monailabel/interfaces/tasks/infer.py", line 300, in run_inferer
network.load_state_dict(model_state_dict)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DynUNet:
Unexpected key(s) in state_dict: "input_block.norm1.weight", "input_block.norm1.bias", "input_block.norm2.weight", "input_block.norm2.bias", "input_block.norm3.weight", "input_block.norm3.bias", "downsamples.0.norm1.weight", "downsamples.0.norm1.bias", "downsamples.0.norm2.weight", "downsamples.0.norm2.bias", "downsamples.0.norm3.weight", "downsamples.0.norm3.bias", "downsamples.1.norm1.weight", "downsamples.1.norm1.bias", "downsamples.1.norm2.weight", "downsamples.1.norm2.bias", "downsamples.1.norm3.weight", "downsamples.1.norm3.bias", "downsamples.2.norm1.weight", "downsamples.2.norm1.bias", "downsamples.2.norm2.weight", "downsamples.2.norm2.bias", "downsamples.2.norm3.weight", "downsamples.2.norm3.bias", "downsamples.3.norm1.weight", "downsamples.3.norm1.bias", "downsamples.3.norm2.weight", "downsamples.3.norm2.bias", "downsamples.3.norm3.weight", "downsamples.3.norm3.bias", "bottleneck.norm1.weight", "bottleneck.norm1.bias", "bottleneck.norm2.weight", "bottleneck.norm2.bias", "bottleneck.norm3.weight", "bottleneck.norm3.bias", "upsamples.0.conv_block.norm1.weight", "upsamples.0.conv_block.norm1.bias", "upsamples.0.conv_block.norm2.weight", "upsamples.0.conv_block.norm2.bias", "upsamples.1.conv_block.norm1.weight", "upsamples.1.conv_block.norm1.bias", "upsamples.1.conv_block.norm2.weight", "upsamples.1.conv_block.norm2.bias", "upsamples.2.conv_block.norm1.weight", "upsamples.2.conv_block.norm1.bias", "upsamples.2.conv_block.norm2.weight", "upsamples.2.conv_block.norm2.bias", "upsamples.3.conv_block.norm1.weight", "upsamples.3.conv_block.norm1.bias", "upsamples.3.conv_block.norm2.weight", "upsamples.3.conv_block.norm2.bias", "upsamples.4.conv_block.norm1.weight", "upsamples.4.conv_block.norm1.bias", "upsamples.4.conv_block.norm2.weight", "upsamples.4.conv_block.norm2.bias", "skip_layers.downsample.norm1.weight", "skip_layers.downsample.norm1.bias", "skip_layers.downsample.norm2.weight", "skip_layers.downsample.norm2.bias", "skip_layers.downsample.norm3.weight", "skip_layers.downsample.norm3.bias", "skip_layers.upsample.conv_block.norm1.weight", "skip_layers.upsample.conv_block.norm1.bias", "skip_layers.upsample.conv_block.norm2.weight", "skip_layers.upsample.conv_block.norm2.bias", "skip_layers.next_layer.downsample.norm1.weight", "skip_layers.next_layer.downsample.norm1.bias", "skip_layers.next_layer.downsample.norm2.weight", "skip_layers.next_layer.downsample.norm2.bias", "skip_layers.next_layer.downsample.norm3.weight", "skip_layers.next_layer.downsample.norm3.bias", "skip_layers.next_layer.upsample.conv_block.norm1.weight", "skip_layers.next_layer.upsample.conv_block.norm1.bias", "skip_layers.next_layer.upsample.conv_block.norm2.weight", "skip_layers.next_layer.upsample.conv_block.norm2.bias", "skip_layers.next_layer.next_layer.downsample.norm1.weight", "skip_layers.next_layer.next_layer.downsample.norm1.bias", "skip_layers.next_layer.next_layer.downsample.norm2.weight", "skip_layers.next_layer.next_layer.downsample.norm2.bias", "skip_layers.next_layer.next_layer.downsample.norm3.weight", "skip_layers.next_layer.next_layer.downsample.norm3.bias", "skip_layers.next_layer.next_layer.upsample.conv_block.norm1.weight", "skip_layers.next_layer.next_layer.upsample.conv_block.norm1.bias", "skip_layers.next_layer.next_layer.upsample.conv_block.norm2.weight", "skip_layers.next_layer.next_layer.upsample.conv_block.norm2.bias", "skip_layers.next_layer.next_layer.next_layer.downsample.norm1.weight", "skip_layers.next_layer.next_layer.next_layer.downsample.norm1.bias", "skip_layers.next_layer.next_layer.next_layer.downsample.norm2.weight", "skip_layers.next_layer.next_layer.next_layer.downsample.norm2.bias", "skip_layers.next_layer.next_layer.next_layer.downsample.norm3.weight", "skip_layers.next_layer.next_layer.next_layer.downsample.norm3.bias", "skip_layers.next_layer.next_layer.next_layer.upsample.conv_block.norm1.weight", "skip_layers.next_layer.next_layer.next_layer.upsample.conv_block.norm1.bias", "skip_layers.next_layer.next_layer.next_layer.upsample.conv_block.norm2.weight", "skip_layers.next_layer.next_layer.next_layer.upsample.conv_block.norm2.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.downsample.norm1.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.downsample.norm1.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.downsample.norm2.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.downsample.norm2.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.downsample.norm3.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.downsample.norm3.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.upsample.conv_block.norm1.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.upsample.conv_block.norm1.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.upsample.conv_block.norm2.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.upsample.conv_block.norm2.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.next_layer.norm1.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.next_layer.norm1.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.next_layer.norm2.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.next_layer.norm2.bias", "skip_layers.next_layer.next_layer.next_layer.next_layer.next_layer.norm3.weight", "skip_layers.next_layer.next_layer.next_layer.next_layer.next_layer.norm3.bias".

@SachidanandAlle @wyli

[wyli]

@yiheng-wang-nv any idea?

[Nic-Ma]

Hi @diazandr3s ,

Thanks for your issue report.
Are you using the latest MONAI nightly docker? Or just install MONAI 0.6RC in your env?

Thanks.

[diazandr3s]

Hi @Nic-Ma, thanks for your help with this. I installed monai==0.6.0rc1 in a new virtual env I created to test it.

[wyli]

perhaps it's the model you trained with v0.5.3 is not compatible with v0.6 because of this change Project-MONAI/MONAI#2166, see also Project-MONAI/MONAI#2158

[diazandr3s]

That might be the reason. I just tried other Apps that don't use the DynUnet and they seem to work. Is there a way to make those weights compatible with the new version? or should I just retrain the models that use DynUnet?

[wyli]

retraining would be the easiest... if you really couldn't retrain it, there is a copy_model_state function, to load previous checkpoint into a new network instance with an additional variable name mapping option:
https://github.com/Project-MONAI/MONAI/blob/42a11259aa8a24d5e83d3b1a493c5a6102282d1f/monai/networks/utils.py#L318-L325

[diazandr3s]

I tried to retrain one model and I've got this error

Traceback (most recent call last):
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/adp20local/anaconda3/envs/monai06_env/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/adp20local/Downloads/MONAILabel/monailabel/utils/others/app_utils.py", line 89, in
run_main()
File "/home/adp20local/Downloads/MONAILabel/monailabel/utils/others/app_utils.py", line 74, in run_main
result = a.train(request)
File "/home/adp20local/Downloads/MONAILabel/sample-apps/deepedit_left_atrium/main.py", line 113, in train
return task()
File "/home/adp20local/Downloads/MONAILabel/monailabel/interfaces/tasks/train.py", line 217, in call
self.trainer().run()
File "/home/adp20local/Downloads/MONAILabel/monailabel/interfaces/tasks/train.py", line 209, in trainer
train_handlers=self.train_handlers(),
File "/home/adp20local/Downloads/MONAILabel/monailabel/utils/train/basic_train.py", line 170, in train_handlers
e = self.evaluator()
File "/home/adp20local/Downloads/MONAILabel/monailabel/interfaces/tasks/train.py", line 192, in evaluator
event_names=self.event_names(),
TypeError: init() got an unexpected keyword argument 'post_transform'

Will investigate more whether this is related to MONAI evaluator or MONAI Label endpoints.

[wyli]

@yiheng-wang-nv is working on the tutorials for the new API, perhaps this ongoing fix (acceleration/distributed_training/unet_evaluation_workflows.py) could be a reference: https://github.com/Project-MONAI/tutorials/pull/246/files#diff-dfe247875ecbd901b66011f0350feca21969f5fdca6eac28a8c2f91329b83b2b