561 497 Provide metrics base APIs by Nic-Ma · Pull Request #2291 · Project-MONAI/MONAI

Nic-Ma · 2021-06-01T15:18:00Z

Description

This PR added base class for the metrics.
And add support for list of channel-first Tensor.
It's a non-breaking change.

Status

Ready

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

merge master

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2021-06-02T11:31:24Z

/black

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Nic-Ma · 2021-06-02T11:34:41Z

Hi @vfdev-5 ,

This PR added the metric base class and support a list of tensors. Tried to align with your new metric API design.

Thanks.

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2021-06-02T15:50:18Z

/black

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2021-06-02T16:39:13Z

/black

Nic-Ma · 2021-06-02T16:48:52Z

/integration-test

Nic-Ma · 2021-06-02T16:49:37Z

Hi @wyli @rijobro @ericspod ,

This PR is ready for review now.

Thanks.

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2021-06-02T22:25:52Z

/black

yiheng-wang-nv

Thanks for the changes, I left some comments above.

wyli

thanks, I put some initial comments, the metric base classes may need more work, let me check the evaluation working group discussions and think about this

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2021-06-03T14:58:08Z

/black

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma · 2021-06-03T16:00:39Z

/black

Nic-Ma · 2021-06-06T00:26:34Z

As this is a very big feature update, I created a new branch: feature/561-metrics-api in the Project-MONAI repo.
I will merge this PR into that branch and create a new PR to dev branch.

Thanks.

* 561 497 Provide metrics base APIs (#2291) * [DLMED] add metric base class Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update meandice and auc Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] extract reduce API Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update regression metrics Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update all the other metrics and enhance unit tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add doc-strings and update unit tests Signed-off-by: Nic Ma <nma@nvidia.com> * [MONAI] python code formatting Signed-off-by: monai-bot <monai.miccai2019@gmail.com> * [DLMED] fix flake8 issue Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix pytype issue Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix all the mypy issues Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix integration test Signed-off-by: Nic Ma <nma@nvidia.com> * [MONAI] python code formatting Signed-off-by: monai-bot <monai.miccai2019@gmail.com> * [DLMED] fix flake8 issue Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix pytype issue Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add more sanity check Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to Yiheng's comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to Wenqi's comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] change to "_compute()" and "aggregate()" Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add compute_list() Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix flake8 issue Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix flake8 issue Signed-off-by: Nic Ma <nma@nvidia.com> Co-authored-by: monai-bot <monai.miccai2019@gmail.com> Co-authored-by: Yiheng Wang <68361391+yiheng-wang-nv@users.noreply.github.com> * [DLMED] update according to comments 1. re-define base classes in 3 levels to make it more clear and flexible to extend 2. added self-contained variables in metrics to provide easier API 3. added reset() and sync() logic 4. added distributed data parallel logic for all the metrics Signed-off-by: Nic Ma <nma@nvidia.com> * [MONAI] python code formatting Signed-off-by: monai-bot <monai.miccai2019@gmail.com> * [DLMED] fix flake8 issue Signed-off-by: Nic Ma <nma@nvidia.com> * [MONAI] python code formatting Signed-off-by: monai-bot <monai.miccai2019@gmail.com> * [DLMED] optimize base classes Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] simplify integration tests to use new APIs Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update based on comments Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] add synced=False in add() Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> * full ci/cd for feature branches Signed-off-by: Wenqi Li <wenqil@nvidia.com> * [DLMED] update aggregate() Signed-off-by: Nic Ma <nma@nvidia.com> Co-authored-by: monai-bot <monai.miccai2019@gmail.com> Co-authored-by: Yiheng Wang <68361391+yiheng-wang-nv@users.noreply.github.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com>

Nic-Ma and others added 12 commits February 1, 2021 19:15

Merge pull request #19 from Project-MONAI/master

42a45e0

merge master

Merge pull request #32 from Project-MONAI/master

cd16a13

merge master

Merge pull request #117 from Project-MONAI/dev

1449a0c

merge master

[DLMED] add metric base class

301b8e5

Signed-off-by: Nic Ma <nma@nvidia.com>

Merge branch 'dev' into 561-provide-metrics-base

c6ad418

[DLMED] update meandice and auc

4f63d2f

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] extract reduce API

9571f53

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] update regression metrics

c31d1ae

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] update all the other metrics and enhance unit tests

5871957

Signed-off-by: Nic Ma <nma@nvidia.com>

Merge branch 'dev' into 561-provide-metrics-base

bb4b5c7

[DLMED] add doc-strings and update unit tests

566a3cb

Signed-off-by: Nic Ma <nma@nvidia.com>

Merge branch 'dev' into 561-provide-metrics-base

e808466

Nic-Ma marked this pull request as ready for review June 2, 2021 11:31

Nic-Ma changed the title ~~[WIP] 561 497 Provide metrics base APIs~~ 561 497 Provide metrics base APIs Jun 2, 2021

Nic-Ma requested review from ericspod, rijobro, vfdev-5, wyli and yiheng-wang-nv June 2, 2021 11:32

[MONAI] python code formatting

0b35531

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

vfdev-5 reviewed Jun 2, 2021

View reviewed changes

Comment thread monai/config/type_definitions.py Outdated

Comment thread monai/metrics/metric.py

Nic-Ma added 4 commits June 2, 2021 21:00

[DLMED] fix flake8 issue

29b87e0

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] fix pytype issue

9f9139c

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] fix all the mypy issues

f07413f

Signed-off-by: Nic Ma <nma@nvidia.com>

Merge branch 'dev' into 561-provide-metrics-base

88f56d4

[DLMED] fix integration test

1372133

Signed-off-by: Nic Ma <nma@nvidia.com>

Nic-Ma force-pushed the 561-provide-metrics-base branch from 9f3945f to 1372133 Compare June 2, 2021 15:59

Nic-Ma added 2 commits June 3, 2021 00:35

[DLMED] fix pytype issue

1a173fb

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] add more sanity check

af858e4

Signed-off-by: Nic Ma <nma@nvidia.com>

[DLMED] update according to comments

e4d5f7b

Signed-off-by: Nic Ma <nma@nvidia.com>