Test time augmentations

[rijobro]

Test time augmentations.

closes #718

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[rijobro]

requires #1795.

[ericspod]

This class will only work with segmentation networks then? Could it theoretically make sense for other tasks like image-to-image transformation? The returned values may make less sense perhaps.

[rijobro]

I've removed the check. We have the return_full_data if the user doesn't want to use the default metrics (e.g., computing the mode probably doesn't make sense for floating point numbers).

[rijobro]

@wyli do you think you could provide some insight into this error? I can't see how any of the files I've modified could cause this.

Also I'm getting a flake error. This comes from:

    if isinstance(transform, Compose):
        # call recursively for each sub-transform
        return any(is_transform_rand_invertible(t) for t in transform.transforms)

When the if statement is satisfied, transform must therefore be of type Compose and therefore has the attribute .transforms. I thought flake was smart enough to figure this out? It does locally, at least.

[wyli]

Hi @rijobro for the first error, I think it's because monai.data is now depending on ignite, this is unexpected (the user should be able to use monai.data without installing ignite).

to replicate the issue in a system without ignite:

>>> import monai.data
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/monai/monai/data/__init__.py", line 37, in <module>
    from .test_time_augmentation import TestTimeAugmentation
  File "/opt/monai/monai/data/test_time_augmentation.py", line 21, in <module>
    from monai.engines.utils import CommonKeys
  File "/opt/monai/monai/engines/__init__.py", line 12, in <module>
    from .evaluator import EnsembleEvaluator, Evaluator, SupervisedEvaluator
  File "/opt/monai/monai/engines/evaluator.py", line 17, in <module>
    from monai.engines.utils import CommonKeys as Keys
  File "/opt/monai/monai/engines/utils.py", line 33, in <module>
    class IterationEvents(EventEnum):
  File "/opt/monai/monai/utils/module.py", line 232, in __getattr__
    raise self._exception
  File "/opt/monai/monai/utils/module.py", line 191, in optional_import
    pkg = __import__(module)  # top level module
monai.utils.module.OptionalImportError: from ignite.engine import EventEnum (No module named 'ignite').

[wyli]

I couldn't replicate the pytype error either... perhaps have a # type: ignore?

[rijobro]

@wyli ok so the ignite must be because I'm importing CommonKeys which sits in engines. It should probably be moved to enums.py so that all modules can access it.

[wyli]

@wyli ok so the ignite must be because I'm importing CommonKeys which sits in engines. It should probably be moved to enums.py so that all modules can access it.

agreed, I think they are generic properties. cc @Nic-Ma this would be a breaking change (not a breaking change by importing the variables)

[rijobro]

Hi @wyli, current tests are failing: https://github.com/Project-MONAI/MONAI/pull/1794/checks?check_run_id=2169625488#step:11:5093. But device is set with device = "cuda" if torch.cuda.is_available() else "cpu". Any idea what's going on?

[wyli]

Hi @wyli, current tests are failing: https://github.com/Project-MONAI/MONAI/pull/1794/checks?check_run_id=2169625488#step:11:5093. But device is set with device = "cuda" if torch.cuda.is_available() else "cpu". Any idea what's going on?

strange, how do I get the corresponding code of that log?

[rijobro]

Don't think you can do it from the logs, but you can obviously get it from the changed files in the PR. Here's the problematic test_testtimeaugmentations.py: https://github.com/Project-MONAI/MONAI/blob/e02a67ce4f94a81027d7006976aea2a5601e92d4/tests/test_testtimeaugmentation.py

[wyli]

Don't think you can do it from the logs, but you can obviously get it from the changed files in the PR. Here's the problematic test_testtimeaugmentations.py: https://github.com/Project-MONAI/MONAI/blob/e02a67ce4f94a81027d7006976aea2a5601e92d4/tests/test_testtimeaugmentation.py

the log corresponds to 6b488c https://github.com/Project-MONAI/MONAI/pull/1794/checks?check_run_id=2169625488#step:3:472 which is the commit before this device update, and the latest test doesn't show the error, correct?

I think the error is addressed by e02a67c e02a67c

[wyli]

thanks!

[wyli]

this could be a warning because the random factors may be in the model, we may want to do test aug with a randomised model...