Skip to content

tests.test_spatial_resample error #4432

Description

@wyli

Describe the bug
randomly run into this issue:

[2022-06-01T14:46:36.804Z] ======================================================================
[2022-06-01T14:46:36.804Z] ERROR: test_ill_affine_1 (tests.test_spatial_resample.TestSpatialResample)
[2022-06-01T14:46:36.804Z] ----------------------------------------------------------------------
[2022-06-01T14:46:36.804Z] Traceback (most recent call last):
[2022-06-01T14:46:36.804Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/transforms/spatial/array.py", line 274, in __call__
[2022-06-01T14:46:36.804Z]     if pytorch_after(1, 8, 0)
[2022-06-01T14:46:36.804Z] RuntimeError: CUDA error: an illegal memory access was encountered
[2022-06-01T14:46:36.804Z] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[2022-06-01T14:46:36.804Z] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[2022-06-01T14:46:36.804Z] 
[2022-06-01T14:46:36.804Z] During handling of the above exception, another exception occurred:
[2022-06-01T14:46:36.804Z] 
[2022-06-01T14:46:36.804Z] Traceback (most recent call last):
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
[2022-06-01T14:46:36.804Z]     return func(*(a + p.args), **p.kwargs)
[2022-06-01T14:46:36.804Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_spatial_resample.py", line 161, in test_ill_affine
[2022-06-01T14:46:36.804Z]     SpatialResample()(img=img, dst_affine=dst_affine)
[2022-06-01T14:46:36.804Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/utils/deprecate_utils.py", line 217, in _wrapper
[2022-06-01T14:46:36.804Z]     return func(*args, **kwargs)
[2022-06-01T14:46:36.804Z]   File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/transforms/spatial/array.py", line 278, in __call__
[2022-06-01T14:46:36.804Z]     raise ValueError(f"src affine is not invertible: {src_affine_}") from e
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 628, in __format__
[2022-06-01T14:46:36.804Z]     return object.__format__(self, format_spec)
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 305, in __repr__
[2022-06-01T14:46:36.804Z]     return torch._tensor_str._str(self)
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 434, in _str
[2022-06-01T14:46:36.804Z]     return _str_intern(self)
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 409, in _str_intern
[2022-06-01T14:46:36.804Z]     tensor_str = _tensor_str(self, indent)
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 264, in _tensor_str
[2022-06-01T14:46:36.804Z]     formatter = _Formatter(get_summarized_data(self) if summarize else self)
[2022-06-01T14:46:36.804Z]   File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 100, in __init__
[2022-06-01T14:46:36.804Z]     nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
[2022-06-01T14:46:36.804Z] RuntimeError: CUDA error: an illegal memory access was encountered
[2022-06-01T14:46:36.804Z] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[2022-06-01T14:46:36.804Z] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[2022-06-01T14:46:36.804Z] 

xform = (
torch.linalg.solve(src_affine_, dst_affine)
if pytorch_after(1, 8, 0)
else torch.solve(dst_affine, src_affine_).solution # type: ignore
)

torch 1.10.2, cuda 10.2

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions