Skip to content

1871 thread-safety for the cache-based datasets#2227

Merged
wyli merged 14 commits into
Project-MONAI:devfrom
wyli:1871-cachedataset
May 21, 2021
Merged

1871 thread-safety for the cache-based datasets#2227
wyli merged 14 commits into
Project-MONAI:devfrom
wyli:1871-cachedataset

Conversation

@wyli

@wyli wyli commented May 20, 2021

Copy link
Copy Markdown
Contributor

Fixes #1871

Description

this PR addresses cache-based dataset multi-thread preloading issues by:

  • adding a monai.transforms.ThreadUnsafe API, the unsafe transforms will be deepcopied in the cache computation.
  • revise all transforms to be either thread-safe or inheriting ThreadUnsafe
  • updated persistent dataset although it's not using multi-threading, mainly to keep the behaviour consistent with the cache-based ones

all the randomisables are not thread-safe, but we only use them with multi-process dataloader, there's no multi-thread issue.

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@wyli wyli requested review from Nic-Ma, ericspod and rijobro May 20, 2021 23:34
@wyli

wyli commented May 20, 2021

Copy link
Copy Markdown
Contributor Author

/black
/integration-test
/black

@wyli wyli force-pushed the 1871-cachedataset branch 3 times, most recently from 5e50a9a to 702ab37 Compare May 21, 2021 10:29
Comment thread monai/transforms/croppad/array.py Outdated
@wyli wyli force-pushed the 1871-cachedataset branch from 702ab37 to c1a6c25 Compare May 21, 2021 12:35
@Nic-Ma

Nic-Ma commented May 21, 2021

Copy link
Copy Markdown
Contributor

Thanks for the enhancement for CacheDataset, it's heavily used in NVIDIA Clara project, I will try to review this PR as soon as possible.

wyli added 12 commits May 21, 2021 15:57
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli wyli force-pushed the 1871-cachedataset branch from c1a6c25 to 90db79d Compare May 21, 2021 14:58

@Nic-Ma Nic-Ma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the enhancement, put several comments inline.
Others look good to me.

Thanks.

Comment thread monai/transforms/transform.py
Comment thread monai/data/dataset.py
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
@wyli wyli enabled auto-merge (squash) May 21, 2021 16:28
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli wyli merged commit a49a63c into Project-MONAI:dev May 21, 2021
@wyli wyli deleted the 1871-cachedataset branch May 21, 2021 23:30
wyli added a commit that referenced this pull request May 26, 2021
* thread safety impl.

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes croppad

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixed intensity

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes spatial

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes util

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* autofix

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* update docstring

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* update datastats

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes docstring

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes torch 1.7 issue

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes mac/win n_workers

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* update based on comments

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* [MONAI] python code formatting

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Co-authored-by: monai-bot <monai.miccai2019@gmail.com>
wyli added a commit that referenced this pull request May 26, 2021
* thread safety impl.

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes croppad

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixed intensity

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes spatial

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes util

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* autofix

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* update docstring

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* update datastats

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes docstring

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes torch 1.7 issue

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* fixes mac/win n_workers

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* update based on comments

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

* [MONAI] python code formatting

Signed-off-by: monai-bot <monai.miccai2019@gmail.com>

Co-authored-by: monai-bot <monai.miccai2019@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cachedataset useability

4 participants